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PREFACE 



this report draws heavily from previous project reports written 
by the original project staff* Also recognition shoujd be given to 
the adjunct professors who gave their tinie and effort to respond to 
the Interviews. 
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PART D - REPORT ON IMPLEMENTATION OF 
DECISON-ORIENTED EVALUATION ^ 
UNITS IN CONSORTIUM AGENCIES 

Thts document ts divided into three main parts* The first part 
IS the introduction^ history^ objectives and rationale for the project* 
Part two (Jeals with the Individual consortium responses to the ques- 
tionnaire developed for this reporrN^e f7na} section contains the 
summary and conclusions derived from the preyldus section* 

tNTROOUCtlON 

Early In the planning process the consortium established the task 

• * 
of developing and implementing a Hodel Training Program which would 

satisfy six main conditions-: 

1) Respond to critical manpower needs in the general area of 
progr^nmatic educational Improvement 

2) Provide experience-based training 

3) Use resources available within the consortium for Implementation 
of the program 

k) Be open-ended and continually responsive to emergent training 
needs In educational Improvement areas 

5) Be guided by a continuing productive evaluation system 

6) Be governed and implemented by the total consortium* 

The membership of the Hodel Training Program Included two project 
Administration agencies, one university-based training agency, eight 
field-based training agencies, five support system agencies, and eight 
cooperating agencies* 

Due to internal (university based) problems, a change i.n leadership 
and a change in focus, N*I*E* decided to phase out the project* It Is the 



purpose of this paper to document the role the consortium played in the 
project^ the results obtained^ and the positive and negative outcomes 
of the consortium involvement, 

CONSORTIUM ORGANIZATION AND RELATIONSHIPS 

This section describes the efforts to develop a functional consortium. 
Included are the structure of the consortium, involvement and commitments 
of its member agencies, principles for its operation, procedu^^es followed 
in its development, and a discussion of how It operated the training com* 
ponent of the overall program. 

It Is to be emphasized that the building of a well fuhctfonlng con* 
sortlum IS a pervasive necessity for the development and operation of the 
proposed training program* The consortium was viewed as the key to the 
following critical concerns: 

1. Bridging the theory^-practlce gap 

2. Making universlty^-based experience realistic 

3. Making field internship experiences meaningful and effective 
learning experiences 

'f. Estal>nsh7ng aTvTable foundation on which a long lasting 

pro9ram can be built which also can ultimately produce large 
numbers of trained people for areas of need . . 

Member Institutions were those field and university-based agencies 
which shared systematically In responsibility and resources for the con*- 
sortium program, and support system agencies shared on an 3d hoc basis* 
Representatives of the member and support system Institutions served as 
the declsion^making team for the consortium* 

Cooperating organizations were those brought in from time to time for 
a specific purpose or project; they would not necessarily be represented 
on the decision-making team* However, if decided by the decision making 
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Figure I 



team that a person from a cooperattng organization would add strength to 
that team, he could be added. 

The program supervisor was the director of the Ohio State University 
Evaluation Center* Directly responsible to him was an executive program 
director, 

the proposed program was comprised of four components: training^ , 
materials development^ dissemination^ and evaluation* Each component 
required a director and staff to be quartered at The Ohio State University 
Evaluation Center* The training component was given first priority for 



staff and f undlng^^gLbreakout of the organization of this component is 



suggests interaction between them and the unWersI ty-^based training agencies 
The third row contains the support system and cooperating organization in 
a similar relationship* ' ^ 

Figure 1 contains the original list of the field-^based training agencle 
and the support system and cooperating agencies* This list was modified 



and changed over the life of the project. The following list Is the most 
current: 





The second row of Figure I shows fleld-^based training agencies and- 



Dr* Henry M* Brickel 1 
Institute for Educational Dev* 
52 Vanderbilt Avenue 
New York, New York lOOt? 



Dr* Jean Butman, Director 
Research and Evaluation Div* 
Northwest Regional Lab 
^00 Lindsay Bui tdtng 
710 S*W* Second Avenue 
Portland, Oregon 9720^ 
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''Or. wnUam J. Gephart 
Director of Research Services 
Phi Delta Kappa 
Eighth Street & Union Avenue 
Bloomlngton^ Indiana 474DI 

-Dr. James Jacobs 
Director of Research 
Cincinnati PubJic Schools 
23D East Ninth Avenue 
CTfncinnatl, Dhio k$202 

Dr. C. Philip Kearney 

Assoc. Superintendent for Research 

and School Administration 
State Department of Education 
Lansing, Michigan kS902 

-Dr. J. A. LasManIs 
Director of Research 
The Council of the Great City 

Schools 
i8ic« -H*' Street, N.W. 
Washington, D.C. 2DDD6 

Dr. Wi 1 1 lam Loadman 
Nisonger Center for MentaJ 

Retardation 
The Dhio State University. 
1S8D Cannon Dr., McCampbell Hall 
Columbus, Ohio 43215 



-Or. Daniel Stufflebeam , 
Director Evaluation Center 
The Dhio State University 
1712 Neil Avenue 
Dxley Hall 

Columbus, Ohio 4321D 

-^Dr. Jack P, Taylor, Sup. 
Saginaw Public Schools 
55D Millard Street 
Saginaw, Michigan 4d6lD 

^^Dr. Jerry Walker 
The Center for Voc. S Tech Ed. 
The Ohio State University 
I960 Kenny Road 
Columbus, Dhio 

**Dr. William Wayson 
The Development Faculty 
The Ohio State University 
Ramseyer Hall 
Colunbus, Dhio W2lb 

^r. William Webster 
Assist. Superintendent 
Research ^ Evaluation Unit 
Dallas Indep. School Dist. 
37DD Ross Avenue 
Dallas, Texas 752D4 



AVitae for these agencies are located in Appendix a. 

INVOLVEMENT, COMMITMENT, AND PRINCIPLES 

All member representatives of the consortium were r-'ual voting members 
of the decision^maklng team. The Ohio State University Evaluation Center 
staff's role was to make recommendations about operational obje *"Jves, 
new members, and other business to the members at decision meetings. These 
recommendations were reacted to, changed, improved, and adopted operating 
policy for the project. 

In addition, nach of the* original members accepted a staff advisory 
position for which he was to be paid as a staff member to assist in the 
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planning of the operational program. In these roles Jack Taylor (Saginaw) 
served as chairman of the decision-maktng team and mrked closely with 
Daniel Stufflebeam, the project director; James Jacobs (Cincinnati) served 
as advisor to the context evaluation staff; Robert Hammond (Montana) served 
as advisor to the input evaluation staff; Gerald Bay (IITRI) served as 
advisor to OarreH Root, the executive officer; Donald Sanders (OSO) served 
as advisor to Barbara Redick, the information Specialist; and Robert Taylor 
(OSU) served as adviser to the consortium development team. 

A request was sent to each member agency for a letter of commitment 
concerning the Model Training Program. Each agency representative was 
" requested to describe in his response what he believed the role of his 
agency to be. 

Robin Fcjrquhar (UCEA)» representing a successful consortium operation, 
consulted with the project staff on the operation of the consortium. With 
his advice* the following list of operating principles was developed: 

Principles for Success of a Consortium * 

1) Member institutions should not be in dysfunctional competition 
with one another* at least insofar as consortium*related 
operations are concerned. 

2) Each member institution should have something unique to- 
contribute to the productive operation of the consortium; 
there must be give* as welt as take, by all members. 

3) Participation in the consortium should yield direct and sig- 
nificant benefit to alt member institutions. 

Contribution of member institutions in the consortium shoutd 
complement or supplement one another in a fruitful fashion; 
they should not cancel one arrother out in terms of impact or 
replicate one arrother unnecessarily. Overall, there should be 
an internal balance among contributions and a gestalt effect 
that Is greater than the sum of the parts. 

5) All member institutions should have clearly established, equal, 
and regular opportunities to influence consortium programs and 
pol icies. 

11 



6) The coDSortlun should be a leadership as well as a service 
agency with respect to me^^toer tnstttution operations; It 
!>hould not only respond to immediate needs, but should 
Identify future needs and design programs to respond to 
them* 

7) Mechanisms must be established for fast and easy communication, 
not Just between consortium members and the central agency, but 
also among consortlum^membc'rs* 

8) Approaches must be developed to assess the effectiveness of 
the consortium's program* This means that objectives for the 
consortltim as a consortium must be specified, means of deter* 
mining the degree to which these objectives are achieved m^ist 
be developed, and procedures for adapting consortium operations 
(or changing consortium objectives) on the basis of evaluative 

^> y feedback must be established, 

9) Procedures and criteria for adding Institutions to, or dropping 
th^ from, consot';.:i<m membership must be established* 

10) The relationships between the consortium (In terms of its 

products and processes) and external Individuals and Institu* 
tions must be delineated (with respect to both Input to and 
output from the consortTum)''. 

It) While a core of benefits from consortium membership should 

be received by all participants In common, the value of other 
significant benefits received from the ^consort I urd should be 
directly related to the extent of contribution made to Its 
operations* 

12) Mechanisms should be devls^sd whereby new Individuals In 
member institutions mf^y be educated and socialized with 
respect to the rights, privileges, and responsibilities of 
consortium membership; the history, purposes, and functions 
of the consortium; the policies, processes, and products of 
consortli^ operations; and the structure, governance, and 
supoort mechanisms of the consortlwi* 

Using the letter of commitment from each consortium m^ber and this 

list of operating principles, a subcontract was negotiated with each 

agency during Phase One of the project* 



HISTORY 



Consortium members were recruited from organizations concerned with 
educational progron Improvement through research, development, diffusion. 
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aiid evaluation. They were also recruited because they were presently 
involved In some RDOfi^E effort for program Improvement. Each agency's 
need for training in RDO&E functions, as well as tts capacity to com-* 
plement needs of other members, were criteria used for selection of 
members* 

At the July, 1970, meeting of the decision-making team, tt was 
nanimously decided that a member should withdraw from the consortium at any 
time It was judged that the member's participation was not mutually beneficial 
to the agency and to the project. This judgment could be made by the member 
agenjcy, the decision-making team, or by recommendation of the project 
director* At present, one agency has been dropped and six have been added* 
tt wais believed that meihbership should be open to this degree in order to 
to maintain a strong functioning consortium* 

The consortium membershlpi included three agencies of The Ohio State 
UniversIty"The Evaluation Center, represented by Daniel Stuff lebeam. 
Director; the Educational Development Faculty, represented by WI 1 1 iam Wayson, 
Chairman; and The Center for Vocational and Technical Education, represented 
by Robert Taylor, Director. Rona.Id G* Kavelock, from the Center for Research 
on Utilization of Scientific Knowledge, Institute for Social Research, 
University of Michigan, was a Itiember for the planning phase, local school 
district memberships Include Cincinnati, Ohio; Dallas, Texas; and Saginaw, 
Michison, represented respectively by James Jacobs, Director of Program 
Research and Design; WiUiam Webster, Assistant Superintendent Planning 
Research and Evaluation; and Jack Taylor, Superintendent. The Council of 
the Great City Schools, Washington, D.C* represented by John Hayman, Chicago, 
Illinois, represented by Gerald Bay, Manager of Technology Utilization 
Section; and Phi Delta Kappa International, Bloomlngton, Indiana, represented 
by Win lam Gephart, Director of Research Services* 
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Context Evaluation Findings 

the context evaluation report, which served as the basis For the 
objectives, was designed to assist the consortiun decision makers to 
identify and assign priorities to. training objectives* 

Areas studied included research, development, diffusion, evaluation, 
social analysis, and program management* Original manpower supply and 
demand data collected from the consortiun agencies, from alt but four of 
the regional educational laboratories, and from twenty*seven state depart-* 
ments of education*' Also, many reports. Including those of Clark and 
Hopkins, Sarbadora, the AERA Task Force on Research Training, Teaching 
Research In Oregon, and other literature In the field were collected and 
analyzed* An analysis and synthesis of the data indicated the following 
priority of training and program development needs: I) evaluation, 2) dif- 
fusion, 3) development, k) research, S) social analysis, and 6) administration* 

Further, diffusion development was found to be hindered by the lack 
of a consistent definition of diffusion; research and development were' 
viewed as having less need for new training programs than either evaluation 
or^djffusion; and the needs were even less,, at least In this study. En the 
fields of social analysis and administration* 

Evaluation was suggested as the core area for the purposed training 
program for three primary reasons* First, it was revealed to be the area 
for greatest priority need* Second, it is closely related to the other 
areas In that it provides a source of data for declslon-^makers In research* 
development, diffusion, and administration* And third, manpower shortages 
projected by Clark and Hopkins indicated critical shortages would occur under 
Elementary and Secondary Education Act Titles 1, I), and tV in the area of 
evaluation If training priorities remained unchanged* This need was sub* 
stantiated by Worthen and Sanders In their analysis of the 1968 AERA 
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enipldyment service data that revealed three evaluation vacancies for every 
trained evatuotor. 

Although the precise number of evaluators needed was unknown, It was 
clear that the demand was not being met* Further, this need would rise 
sharply as the concept of and resulting demand for educational accountat I M ty 
deveibps, and evaluator training programs must be developed to meet the 
projected need>* 

the- Major Goals 

Based upon the context evaluation, evaluation was chosen as the top 
priority area jbPd innovation process (diffusion) as the second priority 
area for the projected training program* 

Evaluation was chosen because It Is the area of greatest need and of 
greatest capability in the consortium* Since evaluation pervades every 
area of the change process, concentration on this area should bear relevance 
for the solution of problems relating to research, development, diffusion 
social analysis, and program administration* 

Diffusion (innovation process), the second greatest area of need for 
training, requires the development of new training capabilities In the 
consortium* 

While research and 'development were found to be lesser areas of need, 
and while they arc not suggested for high priority attention in this project, 
it is to be noted that The Ohio State University has for some time operated 
strong training programs In these areas* Also, the Purdue University Title 
IV Research Training Program was to be a cooperating agency of the program . 
The new training program to be focused on evalatlon was to complement the 
existing training programs In research and development* Concrete efforts 
were made to develop cooperative relationships among these programs* 
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Social analysis and administration (the two areas added by the decision- 
making team for' cons tderat ton along with ROO&E) were not Indicated by the 
context evaluation to merit primary attention by the consortium. 

PROJECT PURPOSE AND RATIOMALE 

The formulation of project objectives was a continuing process En 
response to a variety of factors. These Include the results of the develop*- 
mental activities of the project, the suggestions of consortium representatives 
and program staff, evaluative feedback from the Internal evaluation unit, 
evaluative feedback from the external eva luat Ion j)anel , and limitations of 
funds available for the project. This section Identifies the six basic 
objectives for the Model Training Project and describes the general program 
strategy being followed to achieve them. 

Overall Objectives 

K TO CONCEPTUALIZE A GENERAL MODEL FOR TRAINING EDUCATIONAL 
EVALUATORS. THIS MODEL WILL: 

(a) Specify recruitment procedures, application procedures, 
selection procedures, advisory comnlttee procedures, program 
designing procedures, program experience and monitoring pro- 
cedures, program completion procedures, placement procedures, 
and reentry procedures 

(b) Provide well-defined bases for development of procedures, 
documents, and activities necessary to Its Implementation 

(c) Be sufficiently general izable so that It can be installed 

and operated by groups of field and university agencies beyond 
the consortiun operating the present project 

(d) Be general Izable for training of personnel In the areas of 
research, development, and Innovation process. In addition to 
evaluation 

(e) Be Internally consistent 

(f) Be Interpretable to general education audiences 



(9) Suggest criteria for operational testfng 



(h) Qe viable both In terms of funding and the usual academic 
constraints to be found in groups of field and university 
agencies other than those presently involved In the Model 
Training Project 

TO OPERATIONALIZE A GENEAAL MODEL FOR TRAINING EDUCATIONAL 
EVALUATORS. THE MATERIALS ANO SYSTEMS THIS WILL REQUIRE WILL 
INVOLVE: 

(a) Recruitment procedures and materials 

(b) Selection procedures and materials 

(c) Proficiency tests for trainee diagnosis 

(d) Specific curriculum modules pertaining to content of the 
training model that might be transmitted to students via 
correspondence training, field training, or short-term in- 
stitute-type training 

(e) ^ Material for planning students' programs ^ 

(f) Project resource data bank 

(g) Procedures and materials for placement of students, both 
in Internship and regular job situations* 

(h) Cognitive and affective behaviors that students at each 
}eve1 of the training project will be able to exhibit 
following completion of training 

TO IMPLEMENT THE GENERAL MODEL FOR TRAINING EDUCATIONAL EVALUATORS. 
THIS WILL INVOLVE: 

(a) Identifying a sizable pool of potential trainees for the 
Model Training Project 

(b) Assessing qualifications of applicants for participation In 
the Model Training Project to Identify those most likely to 
pursue an evaluation career and perform ably In such a role 

(c) Selecting outstanding applicants for training who aspire to 
become generallst evaluators, directors of evaluation offices, 
professors of evaluation, and evaluatlon^orlented educational 
leaders 

(d) Each student will be heiped to diagnose his training needs* 
project a relevant program of university and field training 
experiences, and to interact systematically with a committee 
of field'- and university-based professors 

(e) Students will participate In specially devised seminars* and 
participate meaningfully In evaluation work experiences 

17 
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(f) Students wilt exhibit desired terminal behaviors and attftudi 
following completion of their training program^ as specified 
in the overall training model 

(g) Students will be placed in satisfying professional positions 
following completion of their training programs 

(h) Student; will have the opportunity to reenter the Modol 
Training Project .for further training beyond a previously 
completed progrdin 

TO TEST THE GENERAL MODEL FOR TRAINING EDUCATIONAL EVALUATORS* THE 
MODEL WILL BE: 

(a) Tested for Its coherence In terms of the extent Its essential 
features conwiunlcate wel 1 with students^ trainers^ field 

"personnel^ and curriculum specialists. 

(b) Subjected to tests of Internal consistency and comprehensiveness^ 
generallzablllty^ and transportability to determine Its con- 
ceptual adequacy 

(c) Assessed In Its operational form to determine the extent alt 
components have been sufficiently operational i zed ^ as well as 
the extent the parts Interact to serve the overall functions 
of the model 

(d) Described and assessed under conditions of Implementation to 
determine Its operating characteristics^ including costs^ ease 
of use^ and adherence to the conceptual model 

t 

(e) Assessed in terms of the skills^ knowledges^ and attitudes of 
students trained through Its use^ as well as In terms of the 
perceptions of those students and persons who Interact with 
them both during and after training 

TO DEVELOP A CONSORTIUM OF FIELD AND UNIVERSITY AGENCIES THAT WILL 
IMPLEMENT THE TRAINING MODEL IDENTIFIED IN OBJECTIVE K THIS 
CONSORTIUM WILL: 

(a) Include agencies that have taken leadership In educational 
evaluation and that represent schools^ state education 
departments, research and development Institutions^ national 
educational agencies^ and university agencies 

(b) Develop a national faculty of highly qualified evaluation 
leaders who can offer both university- and field-based 
training and who are thoroughly acquainted with and involved 
in the development of the Model Training Project 

(c) Be bound together by appropriate legal agreements 

(d) Design and implement a feedback network among consortium members 
that will maintain good coirvnunlcatlon within the consortium and 
among the members of the national evaluation faculty 
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(e) Provide a real-world laboratory for field-based training In 
eva luat ion 

(f) Serve as a mechanism for the recruitment of able students Into 
the field of evaluation 

(g) Serve as a flrst*llne network of agencies for placing trained 
eva I ua tors 

(h) Provide sites for field testing of the general training model 
and specific training materials 

(i) Provide a network of agencies that accept support and Implement 
effective evaluation procedures 

(j) Develop and implement long-range plans for collaborative efforts 
of the consortium agencies In the Improvement of evaluation 
theory^ practice^ and training \ - 

TO ASSIST AGENCIES OF THE CONSORTIUM TO INSTITUTIONALIZE EVALUATION 
THROUGH PLACEMENT OF MODEL TRAINING PROJECT GRADUATES AND THROUGH 
INSERVICE TRAINING ACTIVITIES IN THE CONSORTIUM AGENCIES* SUCCESS- 
FUL irfSTITUTIONALIZATION EFFORTS WILL BE CHARACTERIZED BY THE 
ABILITY OF THE MODEL TRAINING PROJECT TO: 

(a) Meet evaluation core staff needs of the evaluation system tn 
the consortium agencies 

(b) Train additional on-site personnel by means of an Inservlce 
evaluation training program to develop and Increase the 
evaluation skills of persons presently employed at the con-, 
sortlum agencies 

(c) Assist consortium agencies to diagpose and respond to their 
evaluation needs and problems 

(d) Assess the movement of each consortium agency toward Institu- 
tionalization of evaluation 



General Program Strategy 

The general program strategy of the Model Training Project to accomplish 
the SIX objectives to train evaluators and those In evaluator-related positions 
was through a consortium of university and field-based training sites* Because 
the six objectives are Interrelated^ a complete program requires the accomplish- 
ment of al 1 SIX* 

The consortium of diversified types of educational agencies Influenced 
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the design of the training model because*of the varied training needs of 
the personnel of these agencies. The training model was designed to meet 
the specific needs of these agencies and agencies of a like nature. 

the strategy was to provide, by the end of the project, examples of 
e\/aluatton Institutionalized In a number of different kinds of educational 
agencies across the country. Further, the strategy envisioned a national 
university concept. Adjunct professors at field ^sltes were accessible to 
students and conveniently located for converting evaluation theory to- 
practice. 

the student support strategy of funding students for work positions 
at the Evaluation Center rather than paying. stipends served several purposes. 
It extended a very limited budget to accomplish the project, provided on- 
the-job 'training in the university setting, afforded the opportunity to 
apply classroom learning Immediately to field situations, and offered a fee 
vjatver for coursework. 

In general, the strategy for students provided; 

1} An exposure to a wide variety of multiple experiences. 

2) For student mobility, not only In geographic area, but also In 
type of evaluation. 

3) A training gi'ound for personnel who wDuld eventually fulfill some 
of the manpower needs of evaluation. 

^) An .irea In which students could fulfill their own needs, through 

the generation of an individualized coursework and Inservice training 
program. 

5) A flexible program, not only for the student, but for the agency 
involved. 

As stated earlier, two of the objectives proposed by the Model Training 
Project related directly to the consortium. These objectives were numbers 
5 and 6. In order to fulfill these stated objectives, "Standards for 
Evaluation" and "Proposed Evaluation Processes" were determined by the 
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project staFF. These sr^ndards and processes are presented. below; 



GENERAL OBJECTIVE 5 . TO OEVELOP A CONSORTIUM OF FIELO ANO UNIVERSITY 
AGENCIES THAT WILL IMPLEMENT THE TEIAINING MOOEL lOENTIFlEO IN OBJECTIVE 1. 



Standards For Evaluation 



Report of Proposed Evaluation 
Processes 



(a) This consortium will include 
agences that have taken leader-^ 
ship in educational evaluation 
and that represent schools^ state 
education de partments ^ research 

•^nd^KjevettSpment Inst t tut+onsi ^ 

national educational agencies^ and 
uhlversltY agencies, 

(b) . This consortium will develop a 

national Faculty of highly 
qualiFied evaluation leadeTT 
who can offer both university- 
and field-based training and 
who are throughly acquainted 
wi th .and Involved In the 
development oF the Model Training 
Proj ect 



(c) This consortium will be bound 
together by appropriate legal 
agreements. 



(d) This consortium will design and 
implement a Feedback network 
among consortium members that will 
maincain good communication wrthin 
the consortium and among the members 
oF the national evaluation faculty. 



K The list oF consortium agencies 
and their Functions wll T be 
analyzed by fhe Evaluation 
Unit using a criteria list to 
determine If.the agencies have 
exercised leadership. 



1, The adjunct professor will 

be compared by the evaluation 
unit relbtive to the Ohio-* 
State University educational 
development criteria For 
FacttI ty sel ectlon . 

2. The attendance and degree oF 
Interactive involvement oF 
adjunct proFessors will be 
documented In the quarterly 
consortium meetings. 

K The project director will be 
asked to produce documenta- 
t ! ona I I ega I ag reemen ts that 
bind the consortium, 

I, The project director will be 
asked to produce evidence 
that an information Feedbaack 
network exists between con* 
sortium members (e,g,^ the 
Newsletter and the position oF 
Consortium Oeveloper), 



2, At the quarterly consortium 
meetings adjunct proFessors 
will be surveyed to document 
inter-consortlum contact^ and. 
and to assess the usefulness 
that contact. And following 
the consortium meeting^ adjunct 
proFessors will be surveyed to 
determine awareness oF the 
main issues and decisions impor- 
tant to consortium operation. 
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Standards for Evaluation 



(e) this consortium will provide a 
real-world laboratory for field- 
based training in evaluation* 



Report of Proposed Evaluation 
Processes 



3* The materials development 
component will be asked to 
produce feedback evidence oh 
the use of cofixnunications 
materials such as the Adjunct 
Professors and Procedures 
Handbook * 

I* Student adv Isory* records will 
be Surveyed to assess if 
students are being placed In 
consortium sites for trathlhg* / 

2* The process evaluation data 

collected under General bbjectiv 
3 (Implementation of t;he Jitodel) 
will be analyzed to assess^ i:he 
consortium provision of real 
world laboratory training 
experiences* ^ 



(f) This consortium will serve a$ a 
.mechanism for the recruitment 

of able students into the field 
of evaluation* 

(g) This consortium will serve as a 
first^line network of agencies 
for placing trained evaluators* 



1* Student records will be surveyed 

to assess the percentage of 

students recruited from withir 

the consortium* 
* 

1* Student records will be sur* 
veyed upon graduation to assess 
the initial work site selected 
by the graduate^ and the ratio 
of consortium to non-consor- 
tium placements will be 
completed* 



2* The follow-up proposed for the 
1973-1974 funding period wilt 
provide data from which the 
ratio of consortium to non-* 
consortium work up to a year 
after graduatipn 11 be 
completed* 
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Standards for Evaluation 



(h) This consortium wItt provide sites 
for field testing of the general 
training model and specific train-* 
ing materials* 




(i/ This consortia will provide a 
n&:wc>rk of agencies that accept 
support and Implement effective 
evaluation procedures. 



Report of Proposed Evaluation 

Processes 
^^^^^^^^^^^^^^ 

I* The materials development 
component will be asked to 
produce evidence that 
materials have been field 
tested in conro ttt I ufp 'Agencies* 

2. The process evaluation data 
gathered under General Object- 
ive (Implementation of the 
model) will be analyzed If 
^ the training model Is being ^ 
tested In consortium agencies* 

1. The consortium membcrs^ wIM, 
be asked to submit written 
policy statements of the 
parent organization that in- 
dicate Support of evaluation 
procedures and evidence that 
the member agent Is fulfilling 
Its part of consortium member"^ 
ship* 

2. The consortium member will be 
asked to submit budget reports 
that reflect support for evalu- 
ation activities* 



(j) This consortium wilt develop ^nd 
implement long-range plans for 
collaborative efforti; of the con- 
sortium agencies in the improvement 
of evaluation theory^ practice and 
training* 



3* The reports of the site visit 
teams to occur during 1971*1972 
will be analyzed relotive to 
the implementation of effective 
evaluation procedures* For 
example, the affect of the 
following variables on decision-* 
making will be assessed: 
staffing, evaluation unit place* 
ment on organization chart, etc* 

^* Institutionalization plans and 
reports will be analyzed relative 
to increesed Implementation and^~ 
effectiveness of-evaluation 
procedures* 

I* The project director will be 

asked for evidence of the followl 

(a) materials and documents 

relative td an on-going con- 
sortium evaluation plan 
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Standards for Evaluation Report of Proposed Evaluation 

Processes 

) (b) Evidence of plans and 

actions of collaboration 
between agencies* 

(c) Evidence that the quarterly 
consortium meetings provide 
for and produce policy 
decisions to establish and 
carry out long-range col* 
loboratlve Inter^agency plant 
to Improve evaluation thearyj 
practice and training* 



GENERAL OBJECTIVE 6. TO ASSIST AGENCIES OF THE CONSORTIUM TO INSTITIJTION 
ALIZE EVALUATION THROUGH PLACEMENT OF MODEL TRAINING PROJECT GRAOWTES 
AND THROUGH INSERVICE TRAINING ACTIVITIES IN THE CONSORTIUM AGENCIES. 



Standards for Evaluation 



(a) Successful Institutional liat Ion 
efforts will be characterized 
by the ability of the Model 
Training Project to meet 
evaluation core staff needs of 
the evaluation system tn the 
conscrtium agencies* 



(b^ " Successful insti tutional Izatlon 
efforts will be characterized by 
the abiMty of the Model Training 
Project to train additional on- 
Site personnel by means of an 
Inservice evaluation training 
program to develop and Increase 
the evaluation sktlls of persons 
presently employed at the''con^ 
sortEwn agencies 



RePOrt of Proposed Evaluation 
Processes 

K The fdllow»up of students 
during the 1973*1974 funding 
period will gather data 
relative to graduate per*- 
formance and agency staff 
nee^s . 

2. Consortium agency adjunct 
professors will be surveyed 
to seek their core. staff 
needs. This data will be 
compared with the quantity 
of graduates and trainees 
available to serve agency 
core needs* 

1. Adjunct professors will be 
asked to produce documented 
evidence attendance figures^ 
(e*g*» agenda^ materials^ 
etc.) in-service training at 
their site* 

2. The materials development 
component will be asked to 
specify and present evidence 
of materials developed for 
Inservlce training. 
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Stondards for Evaluation 



(c) Successful Institutional izat Ion 

efforts win be characterized by the 
ability of the Hoclei Training 
Project to assist consortium agencte: 
to diagnose and respond to their 
evaluation needs and problems. 



(d) Successful institutional izatlon 
efforts wUKbe characterized by 
the ability of the Model Training 
Project to assess the movement 
of each consortium agency toward 
institutionalization of evaluation* 



Report of Proposed Evaluation - 
Process 

3* A record of the use of 
materials development 
materials will be a foll<M- 
up feedback Instrument * 
included with each materials 
package* 

I* The reports from the 1971*1972 
site visit teams wll I be 
analyzed to assess If needs 
and problems have been 
del Ineated* 

2. At the quarterly consortium 
meetings adjunct professors 
will be surveyed relative 
to project aid received in 
the diagnosis of needi and 
problems (e,g,» the usefulness 
of the site visit self- 
evaluation forms^ the effects 
of 0*S*U* sponsore^j mint* 
workshops/etc,) 

I*' The adjunct professor's 

quarterly consortium meeting 
reports on the Institution- 
aUzatlon of evaluation at 
their site will be assessed 
over time to determine the 
Increase or decrease In 
Institutionalization* The 
findings of the site visit 
teams during the I97M972 
funding period will provide 
a standard for confjarlson for 
each agency. 
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CONSORTIUH INTERVIEWS 

This section contains the individual responses of adjunct professors 
t 

to the Interview dues tlonaa Ire* A copy of the questionnaire Is In 
Appendix B* In addition to the questionnaire* each agency was given 
a copy of the Criteria for MTP Consortium Member "^^p* which Is In 
Appendix C* This technique was used primarily to solicit responses that 
would highlight the strengths and weaknesses of a consortium* All 
interviews were done on site by the present 'AT? staff assigned to 
this particular product* 
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CONSORTtUM INTERVIEW 
Dr* Arnold Ashburn 
Dr* WlUiam Webster 
Dallas Independent School District 

May 10 & 11, 1973 
interviewer: John HHderbrand 



Section I 



K. WHAT WERE THE NEEDS OF YOUR ORGANIZATION (RELATE TO): 
A')" PROGRAMS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE: 1) Problems: 

a) Recruiting, training, and upgrading personnel 
with the agency 

b) Di^ not have a good supply of qualified people 
- . to choose from 

c) Difficult to get good people especially at the 
masters and bachelors level 

2) Projects; 

a) District does not htre for a particular project 
(Research & Evaluation section); a person might 
work on three or four different projects. 

3) Organizational: 

I) Strength in belonging to a network^ plus more 
visibility 

2. WHAT PROBLEMS WERE ASSOCIATED WtTH THE NEEDS CITED ABOVE? 

RESPONSE: I) Accentuated the need for staff development; very 
concerned with staff credibility 

2) Recruiting supply of qualified personal was scarce; 
the consortium gave th<f district a pipeline on people 
to be hired and the Interns could be observed before 
hiring; one advantage for the university and students 
was that placement could occur prior to graduation; 
a second advantage to the district was the placing of 
interns into open positions until qualified people 
could be found to occupy these positions* 

3. HOW DID EVALUATtDN EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

RESPONSE: 1) Same as response to question 2 above 
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Section tl 

1. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED VOUR DECISION TO 
BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: 1) Opportunity: notified of the opportunity by a staff 
member from Dallas of the possibility of joining the 
consortium 

Needs: See Section 1 

Problems: Concerned with cost. In terms of money, 
manpower and time allocations 

2. WHAT PROBLEMS DID YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT FROM 
VOUR AGENCY? 

RESPONSE: 1) The main problem was based on the question of legality; 

also, there was some concern of committing the district 
to a project based out of state. 

3. IN REVIEWING THE CRITERIA FOR SELECTION , WHICH CRITERIA WAS THE MOST 
DIFFICULT? WHY? 

RESPONSE: I) Number 7 criteria, "The agency sh^ll commit Itself to 
pay support and to pay tuition costs for all trainees 
receiving Ohio State University credit In the time they 
are based as an Intern In that agenc/*, was the most 
difficult; Number 5 criteria, "The agency shall serve as 
a recruitment and placement agency for trainees of the 
program*" was the second most difficult* 



Section 1 1 1 

1. WHAT ADVANTAGES DID YOU EXPECT TO GAIN AS A MEMBER DF THE CONSORTIUM? 

RESPONSE: 1) Pipeline for hiring, able to observe Interns 

2) interaction and cooperation with other agencies 

3) Training of own staff; the staff of the district able 
to observe Interaction In the quarterly meeting held 
In Dallas 

't) Free exchange of materials, reports, designs 

5) Training materials for both personnel and materials 
disseminated to the consortium prior to publication- 

6) Visibility to the district 

2. WHAT WERE THE AOVANTAGES OF BELONGING TO THE CONSORTIUM? 

RESPONSE: 1) Same vIsIblHty 

2) Able to hire some people 

3) Some training not as much Inservlce training as wanted 
't) Some exchange between consortium especially quarterly 

meeting 



Ik 

Sectioa III (Continued) 

3. WHAT PROBLEMS OID YOU EXPERIENCE AS A RESULT OF BELONGING TO THE 
CONSORTIUM? 

RESPONSE: I) Major problem as seen by the district was no reduction 
of work load for those participating in the consortium; 
therefore, the classes suffered. 

Comment: the on*site training was good; money should 

have been provided for released time to the professionals) 

in the district who participated in the consortium. 

k. TO WHAT EXTENT- WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF THE 
CONSORTIUM? 

A) SHARING OF MATERIALS 

B) WORKING OUT RESOLUTIONS TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: 1) Sharing of materials: not any more so than normal 

2) Working of resolution to common problems! No 

3) Exchange of personnel: No 

5. WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR ORGANIZATION? 
IN WHAT WAY? 

RESPONSE: I) No; not sure of value 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 
RESPONSE: I) No 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH THE MTP's 
EVALUATION UNIT? 

RESPONSE: I) Worked with, Dr. Bunda in setting up: 

a) Plans for processing data (SAES) 

b) Helped revise items for SAES 

c) Scoring of SAES 

d) Plans for computerizing of PRDB 

-J. 

8. IN WHAT WAYS DID THE PRDB SERVE YOUR AGENCY? 
RESPONSE: I) None 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PRDB STAFF? 

RESPONSE: 1) Yes; refer to question iil , Section II'. 

?0. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES WITH 
THE MATERIALS DEVELOPMENT COMPONENT OF THE MODEL TRAINING PROJECT? 

RESPONSE: 1) yes; a) revised the modules on measurement 

b) student handbook and adjunct professor handbook 
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Section tV 

\, TO WHAT EXTENT WERE YOU AKD YOUR AGENCY INVOLVED IK THE PLANKlKG PHASES 
OF THE MTP PROJECT? 

RESPONSE: I) Participated in the planning grant phase 

2) Participated in a decision making mode 

3) ■ Satt sf led wi th openness and Interactions 

2, DID YOU PARTICIPATE \H THE SELECTION OF MEMBERS OF THE CONSORTIUM? 
RESPONSE; I) YeS 

3. WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IK MTP 
DECISION MAKING? 

RESPONSE: 1) Generally satisfied 



Section V 

1. HOW MANY INTERNSHIPS WERE AVAILAELE IN YOUR ORGAKIZATIOK FOR MTP 
STUDENTS? 

RESPONSE; I) Convnitted up to 6 interns, but only had 3 at any one 
time; needed to have a position in order to place 
Interns; tried to hold a position open for this 
purpose. 

2. WHAT PROBLEMS DID YOU FACE IN ACdUlRtNG INTERNSHIPS? 
RESPONSE: I) None 

3. WHAT PROBLEMS DID YOU EXPERIENCE IN REUTIONSHIP TO THE ADVISEMENT OF 
INTERNS? (EXAMPLE: INFORMATION CONCERNING PROBLEMS YOU EXPERIENCED) 
IF ANSWER TO ONE WAS NO, GO TO NEXT SECTION. 

RESPONSE: I) Major problem; not familiar enough With Ohio State 

University to adequately advise; both iludent Handbook 
and Adjurrcc Professor Handbook was helpful; the 
internship coordinator, Dr. Relnhard, was also very 
helpful 

^. WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 
RESPONSE; I) Yes, also, all interns were at the B.A. ^evel 



Section VI 



WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION UNIT AS OF 
DECEMBER 30, 1972? (IF NO BUDGET WAS AVAILABLE, GO TO NEXT SECTION) 



RESPONSE: I) $86l,6itl.OO 
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'Section VI (Continued) 

2. HOW MUCH OF THE MONEY kttS SOFT MONEY? 

RESPONSE: I) $5It5, 000. hard money 
2) $3I6,6'tr soft money 

Section VII 

1. 0!0 YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? (IF NO UNIT 
WAS ESTABLISHEO GO TO NEXT SECTION) 

RESPONSE: I) Already established 

2. TO WHAT EXTENT WAS THE EVALUATION COMPONENT DEVELOPED? 
?<ESPONSE: I) None 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR UNIT? 
RESPONSE: I) 18 Ph.O. and 30 support personnel 

^. WHAT PROBLEMS DID YOU EXPERIENCE IN ESTABLISHING THE UNIT? 
RESPONSE: 1) None 

5. WHAT SOLUTIONS WERE REACHEO? 
RESPONSE; I) None 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT YOUR 
EVALUATION PLAN? 

RESPONSE: I) Exposure to various components of C.I. P. P. 

2) CIPP, especially the Interface role, helped with 
communtty with principals and teachers 

Additional Coniroents 

There should be two Internships for each student. Tlie first, lasting 
two or three days should be primarliy for the purpose of visiting a cross* 
sample of the agencies. This would enable the student to structure a better 
program while at the university. The second Internship should last 
approximately 6 months. If the Internship Is funded by the consortium, an 
intern would receive a variety of experiences. If the agency pays them, 
the Intern would be assigned to a particular posftlon. 
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CONSORTIUM INTERVIEW 
Dr. J.C. LaMants 
Council of G;eat City Schools 

May, 1973 
Interviewer: Bill Nealy 



Section I 

K WHAT WERE THE NEEDS 0F-.<YO0R'bRGANIZATI0N (RELATE TO): 

A) PROGRAMS * 

B) PROJECTS • 

C) ORGAN IZATIOMAL (PIANS) ' 

RESPONSE: I) Assisting evaluation personnel In our cities 

2) Could be a field agent 

3) Had no In-house evaluation procedures 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 
RESPONSE: )) See question I above 

2v HOW 010 EVALUATION EFFECT DECISIONS WHICH RELATED TO THE 
RESOLUTION OF NEEDS AND PROBLEMS? 

RESPONSE: I) The site visit was very valuable; It helped 
to have someone come In and take a look at 
our Operation 

2) We Implemented all of the suggestions that 
we could 

3) The report was used as a partial basis for 
reorganizing operating structure 



Section II 

1. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION 
TO BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: I) Had some structural and operational problems 
2) The Center could provide us with additional 
evaluation capabilities through Its Interns 

2. WHAT PROBLEMS DID YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT 
FROM YOUR AGENCY? 

RESPONSE: I) None 
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Section 11 (Conttnued) 

3. (N REVIEWING THE CRITERIA FOR SELECTION . WHICH CRITERIA WAS 
the" HOST DIFFICULT? WHY? 

RESPONSE: I) AUocating resources to establish an evaluation 
unit 



Section in 

1. WHAT ADVANTAGES DID YOU EXPECt TO GAIN AS A HEHBER OF THE 
CONSORTIUM? 

RESPONSE: I) Council wanted to establish professional ties 

with academic community and Its top profiesslonals 
injthe' f'ibid of evaluation 

2) A^lt {'ourselves of the expertise that the Center 
offered 

3) Internship program 

'f) SuppTy..of top level people 

5) Current awareness of what. Is going on 

2. WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 
RESPONSE: 1) Received a good evaluation of th<» organization 

3. WHAT PROBLEMS C '0 YOU EXPERIENCE AS A RESULT OF BELONGING. TO 
THE CONSORTIUM? 

RESPONSE: 1) None 

'f. TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS 
OF THE CONSORTIUM? 

A) SHARING OF HATER tALS 

B) WORKING OUT RESOLUTIONS TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: 0 None 

5. WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR ORGANIZATION? 
tN WHAT WAY? 

RESPONSE: t) Yes, see Section I above 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 
RESPONSE: 1) No 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH 
THE MTP'S EWHWTION UNIT? 

RESPONSE: I) Worked on SAES 
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Section III fConttnued) 

8. IN WHftT WAYS DID THE PROB SERVE YOUR AGENCY? 
RESPONSE: I) Received materials 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PROB STAFF? 
RESPONSE: 1) NO 

10. WERE YOU OR MEMBERS' OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES 
WITH THE MATERIALS DEVELOPMENT COMPONENT OF THE MODEL TRAINING 
PROJECT? 

RESPONSE: 1) NO 
Section IV 

1. to WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED IN THE PLANNING 
PHASES OF THE MTP PROJECT? 

RESPONSE: I) Not applicable 

2. DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 
RESPONSE: I) Not applicable 

3. WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IN MTP 
DECISION-MAKING? 

RESPONSE: I) Not applicable 
Section V 

1. HOW MANY INTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR MTP 
STUOENTS? 

RESPONSE.- 1) Not applicable 

2. WHAT PROBLEMS DIO YOU FACE IN ACQUIRING INTERNSHIPS? 
RESPONSE: I) Not applicable 

3. WHAT PROBLEMS DID YOU EXPERIENCE IN RELATIONSHIP TO THE ADVISEMENT 
OF INTERNS? 

RESPONSE: 1) Not applicable 
k. WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 
RESPONSE: I) Not applicable 
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Section VI 

1. WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION UNIT AS 
OF DECEMBER, 1972? 

RESPONSE: 1.) Not applicable 

2. HOW MUCH OF THE HONEY WAS SOFT HONEY? 
RESPONSE: 1) Not applicable 

Section VI I 

1. DID YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? 
RESPONSE: 1) Not applicable 

2. TO WHAT EXTENT WAS THE EVALtlATlON COHPONEMT DEVELOPED? 
RESPONSE: 1) Not applicable 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR UNIT? 
RESPONSE: 1) Not applicable 

WHAT PROBLEMS DtO YOU EXPERIENCE IN ESTABLISHING THE UNIT? 

RESPONSE: 0 Not applicable 

S. VHAT SOLUTIONS WERE REACHED? 

KESPONSE: I) Not applicable 

&V HOW MUCH INTERNAL TRAINING VMS NECESSARY IN ORDER TO IMPLEMENT 
YOUR EVALUATION PLAN? 

RESPONSE: I) Not applicable 
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CONSORTIUM INTERVIEW 

Jerry Baker 
Saginaw Public Schools 

May 17, 1973 
Interviewer: Evelyn Brzezlnskl 



(This Interview was conducted with Saginaw's Director of Testing and Evalu- 
ation rather than with Dr. Jack Taylor, Su)>erintendent of Schools, Saginaw's 
off icial* representative to the HTP consortium.) 



Section I 

1. WHATUERE THE NEEDS OF YOUR ORGANIZATION (REUTE TO): 
A) PROGRAMS 

e) PROJECTS 

C) ORGANIZATIONAL (PLANS) 
RESPONSE: None 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 
RESPONSE: None 

3. HOW DID EVALUATION EFFECT DECISIONS WHICH REUTED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

iRESPONSE: None 



Sfection II 

1. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION TO 
BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: 1) An opportunity to continue an association with 

Stufflebeam begun vrfien Taylor was In XenJa, Ohio, 
(prior to his tenure at Saginaw); prior working 
relationship (1967-1970) between the Evaluation 
Center and Saginaw Public Schools after Taylor 
went there 

2) Continuing staff development 

3) Good PR for Saginaw to be associated with Stufflebeam 
and Evaluation Center 

2. WHAT PROBLEMS DID YOU EXPERIENCE IN (lETTING A cETTER OF COMMITMENT FROM 
YOUR AGENCY? 

RESPONSE: 0 No real troubles; only a few concerns, e.g., money and 
time commitments 
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Section II (Continued) 

3. IN REVIEWING THE CRITERIA FOR SELECTION* WHICH WERE THE MOST DIFFICUin 
WHY^ 

RESPONSE: 1) Criteria #3. '#S^, #7; matnly money problems 
Section III 

1. WHAT ADVANTAGES DID YOU EXPECT TO GAIN AS A MEMBER OF THE CONSORTIUM?- 
RESPONSE: }) See response to Section It* item I* above 

2. WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 

RESPONSE: I) Staff development (Baker benefited directly* probably 
1-2 others wuld have had MTP continued) 

2) Exchange of Ideas with other consortium agencies 

3) PR has benefited Saginaw is pretty welf known 
throughout Michigan when it comes to evaluation* 

3. WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT OF BELONGING TO THE 
CONSORTIUM? 

RESPONSE: l) Some problems with communications' from MTP staff; 

for example, It was difficult to get Information on 
the actual mechanics of setting up a course for 
Taylor to teach In Saginaw 

2) Out-of-state tuition charged by Ohio State was a 
problem 

3) Taylor did not always read or pass on to Baker all 
of the Information about the MTP, so Baker was 
sometimes In the dark about what was happening; 
this problem was eventually corrected 

M They were asked by MTP staff and students to coci^lcte 
quite a few questionnaires 

TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF THE 
CONSORTIUM? 

A) SHARING OF MATERIALS 

B) working of RESOLUTION TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: 0 There was some Informal interaction with Cincinnati; Baker 
was a member of the site visit team which visited lED; 
there was some exchange (or at least requests for exchange) 
of materials between Saginaw and other consortium agencies 

5. WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR ORGANIZATION? IN 
WHAT WAY? 

RESPONSE: I) Yes, the site visit acted as a catalyst in a way; helped 
B3ker get things done which he wanted to do; the instru- 
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Section 1 1 1 (Continued) 

ment used during the site vtstt wds good and bad; good tn 
that tt helped Baker visualize what he was buHdtng toward 
tn an evaluation untt, and bad In that It was too long and 
not really too relevant to Saginaw 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 

RESPONSE: 1) Baker was on a team that visited lED; he felt the 
experience was beneficial 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH THE 
MTP*s EVALUATION STAFF? 

RESPONSE: 1) Baker has no knowledge of any involvement* except 
perhaps that they responded to questionnaires the 
unit might have sent 

8. IN WHAT WAYS DID THE PRDB SERVE YOUR AGENCY? 

RESPONSE: 1) It was Saginaw's fault they did not use the PRDB more; 

Baker did not really understand Its operation until 
he was in Columbus (but that Is because he didn't read 
the Information about the PRDB that had been sent to 
Saginaw); Baker was most Interested In evaluation 
reports from other city school systems 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PRDB STAFF? 

RESPONSE: 1) Not to Baker's knowledge 

10. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN' SHARED ACTIVITIES WITH 
THE MATERIALS DEVELOPMENT COMPONENT DF THE MTP? 

RESPONSE: I) Not unless questionnaires were sent to Saginaw from 
that unit 



Section IV 

1. TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED IN THE PLANNING PHASES 
OF THE MTP? 

RESPONSE: 1) Baker wasn't involved at all* but he feels sure that 

Taylor must have been Involved to a considerable extent 
(but he dosen't know how specifically) 

2. DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 

RESPONSE: I) Baker dosen't know of Taylor's involvement in this 
act ivi ty* 
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Section IV (Continued) 

3. -WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IN HT? OECISION- 
V-HAKING? 

RESPONSE: 1) Baker would as^^ume that Taylor was very satisfied, since 
he was chairman of the decisionmaking team 



Sectiofl V 

1. HOW MANY INTERNSHIPS'WERE AVAILABLE IN YOUR ORGANIZATION FOR HTP - 
STUDENTS? 

RESPONSE: Formally, there were none (basically because of a money 
prob'sm). Taylor might have anticipated morale problems 
among other young employees who were working on their own 
time to complete their defirees or who had Jt st received 
their degrees and who might have iresponded negatively to 
the sight of HTP Interns receiving a salary, academic 
credit, etc., for working In Saginaw. Ge.ierally speaking. 
Baker would have liked an Intern for a /ear, not three 
months 

2. WHAT PR08LEHS DID YOU FACE IN ACQUIRING INTERNSHIPS? 
RESPONSE: )) Not applicable 

3. WHAT PR08LEHS DIO YOU EXPERiENCE IN RELATIONSHIP TO THE AOVISEHENT OF 
INTERNS^ (EXAMPLE: INFORMATION CONCERNING PROBLEHS YOU EXPERIENCED) 

RESPONSE: Not applicable 

'f. WERE YOU SATISRIEO WITH THE INTERNS YOU RECEIVED FROM THE HTP? 

RESPONSE: 1) Not applicable 



Section VI 

1. WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION UNIT AS OF 
DECEMBER 30, 1972? 

RESPONSE: 1) Around $200,000 

2, HOW HUCH OF THE MONEY WAS SOFT HONEY? 
RESPONSE: 1) 60% 
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Section VI i 

1. DID YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? 
RESPONSE: l) Such a untt was established !r> fall, 1968 

2. TO UilAT EXTENT WAS THE EVALUATION COMPONENT DEVELOPED? 
RESPONSE: I) None 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR UNIT? 

RESPONSE: 1) It has grown from one professional person (prior to fall, 
*68)to five professional persons (in fait, *68) to 
seven professional persons (in Spring, '7l). 

WflAT PROBLEMS DID YOU EXPERIENCE IN ESTABLISHING THE UNIT? 

RESPONSE: 1) They could not hire really experienced people because 
of the tentativeness of funding (much soft money) 

5. WHAT SOLUTIONS WERE REACHED? 
RESPONSE: 1) None 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT YOUR 
EVALUATION PUN? 

RESPONSE: I) There Is not much training of central office staff; 

Baker just urges people to take courses ct surrounding 
universities; a three-day evaluation workshop was held 

^ for all too administrators In the district about ti 

years ago; there Is almost no Involvement with classroom 
teachers other than in the testing program 
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CONSORTIUM INTERVIEW 

Dr* Jean Butman 
Northwest Regional Lab 

May 1973 
Interviewer: William Beruttt 



Section I 

K WHAT WERE THE HEEDS OF YOUR ORGANIZATION (RELATE TO): 

A) PROGRAMS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE: I) The needs of HWRL with respect to joining the consortium 
was not connected to its organizational setf^develppoieht 
Its needs were to help strengthen its internal on-going 
organization; one need was obtaining additional training 
for Its own personnel and also to obtain Interns from 
other agencies; also« saw It as a means to make contact 
with other agencies concerning evaluation and to see 
what they were doing; these contacts could also be a 
medium to allow KWRL to move Its material^out Into the 
field; also felt consortium would be a gooti outside 
push to foster formal staff training and In^-servlce 
programs; this would Involve establishing courses and 
seminars In conjunction with Ohio State University; 
they were already doing this Informally 

. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 

RESPONSE: I) Setting of courses and seminars in conjunction with 

Ohio State University presented a problem with respect 
to credit procedures and staffing and paying out*of* 
state tuition; obtaining interns presented a problem 
with respect to salary and relocation and avaitablttty 

. HOW DID EVALUATION EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

RESPONSE: I) Felt NWRL had much to say about what RDD€^E was all 

about since KWRL already had an evaluation theory and 
evaluation units were already an integral part of the 
NWRL 
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Section II 

1. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION TO 
BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: t) Butman was not Involved In the conceptual Izatlon of 
NWRL as a consortium agency*. *Ed Seger was director 
at the time and therefore Butman is not sure of exact 
reason for becoming a member; Butman saw It as partly 
political since NWRL already had a strong base In the 
northwest United States they needed 1) a medium to 
reach out and 2) to also see how others dealt with 
evaluation problems 

2. WHAT PROBLEMS OlD VOU EXPERIENCE tN GETTING A LETTER OF COMMITMENT FROM 
YOUR AGENCY? 

RESPONSE: 1) None 

3. IN REVIEWING THE CRITERIA FOR SELECTION W HICH CRITERIA WAS THE MOST 
DIFFICULT? WHY? 

RESPONSE: t) Institutionalizing an evaluation unit was Irrelevant 
since there already was one; had difficulty receiving 
interns because of expenses (relocation and salary); 
also because of the aval 1 abl 1 1 ty of interns when ^{WRL 
needed them; also policy did not permit ^{WRL to pay 
tuition; NWRL had no general fund 



Section 111 

1. WHAT ADVANTAGES DID YOU EXPECT TO GAIN AS A MEMBER OF THE CONSORTIUM? 

RESPONSE: 1) Expected to gain in-service training and Interns; 

introduce and possibly conceptualize the notion of 
a national faculty; also medium for NWRL thinking and 
materials to move out Into the field 

2. WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 

RESPONSE: I) One of the advantages was increased knowledge about 
univeriity education departments; NWRL placed three 
Individuals as interns at Ohio State University who are 
receiving good training; two of them are returning to 
NWRL as full time professionals; when Butman assumed 
role of active director^ much of the personnel tacked 
technical skills (e.g.^ coding^ programming^ and 
statistical knowledge); technical help from Ohio 
State University was good 
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Section Ml (Continued) 

3. WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT OF BELONGING TD THE 
CONSORTIUM? 

RESPONSE: I) None, other than the polittcal conflict which evolved 
between Stufflebeam and the Educational Development 
Department; question of whether the three MTP students 
could finish their degrees 

^. TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF THE 
CONSORTIUM? 

A) SHARING OF MATERIALS 

B) WORKING OUT RESOLUTIONS TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: 1) Other than consortium meetings and a few telephone 
conversations, very tittle exchange occurred. 

5. WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR ORGANIZATION? 
IN WHAT WAY? 



RESPONSE: I) No site visits occurred; scheduling of site vistts was 
a problem; felt site visit wanted to Introduce concept 
of Institutionalization^ but tlWRL was already Involved 
in on-gofng evaluation and consequently saw no mutual 
advantage; also attitude of site^visit was one of lookl 
at consortium and saying this Is what you are doing 
wrong and saying this Is how you should do It rather^ 
than this Is what you know now and where do we go next 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 

RESPONSE: 1) No 



7- TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH THE MTP' 
EVALUATION UNIT? 

RESPONSE: 1) Assisted with work on criteria for skill objectives for 
students; also took considerable amount of time to 
respond to mailed questionnaires from Mary Anne Bunda 

8, IN WHAT WAYS OID THE PRDB SERVE YOUR AGENCY? 

RESPONSE: 1) Used It to find telephone numbers^ addresses^ and In 
keeping track of who was who^ and who was where 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PRDB STAFF? 



RESPONSE: 1) No 
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Section 1 1 I (Continued) 

10. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES WITH 
THt MATERIALS DEVELOPMENT COMPONENT OF THE MODEL TRAINING PROJECT? 

RESPONSE; I) Had discussions with Jack Sanders and Jack came to 

Portland to view NWRL: Butman felt materials develop- 
ment component was involved In development activities 
without much background or asslstaj^ce; she also felt 
whole process of theory to what-ls-golng-on, back to 
theory was handicapped, but they were doing a good 
Job anyway 

Section IV 

1. TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED IN THE PLANNING PHASES 
OF THE MTP PROJECT? 

RESPONSE: 1) None 

2. DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 
RESPONSE: I) After becoming a member, yes 

3. WERE 'YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IN MTP DECISION- 
MAKING? 

RESPONSE: I) Yes 
Section V 

1. HOW MANY INTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR MTP STUDENTS? 
RESPONSE; 1) Periodically several opened up 

2. ,WHAT PROBLEMS DID YOU FACE IN ACQUIRING INTERNSHIPS? 

RESPONSE: 1) Conflict of academic schedules, tyition, and relocation 
costs , 

3. WHAT PROBLEMS DID YOU EXPERIENCE IN RELATIONSHIP TO THE ADVISEMENT OF 
INTERNS? (EXAMPLE, INFORMATION CONCERNING PROBLEMS YOU EXPERIENCED) 
IF ANSWER TO ONE WAS NO, GO TO NEXT SECTION. 

■a 

RESPONSE: I) None '' 

WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 

RESPONSE: I) Did not receive any 
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Section Vt 

1. WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION m\J AS OF 
DECEMBER 30, 1972? {IF NO BUOGET WAS AVAILABLE, GO TO NEXT SECTION) 

RESPONSE: 1) Do not have separate evaluation unit budget; each. unit 
is part of a division and Is incorporated within that 
division's budget; probably approximately 1/3 of 
resources, 

2. HOW MUCH OF THE MONEY WAS SOFT MONEY? 
RESPONSE: I) All was soft money 

Section VI t 

1. DIO YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? {IF NO UNIT 
WAS ESTABLISHEO GO TO NEXT SECTION) 

RESPONSE: t) No, already had one 

2. TO WHAT EXTENT WAS THE EVALUATION COMPONENT OEVELOPED? 

RESPONSE: 1) Shifted over time and not In direction reconmended by MTP 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR UNIT? 

• RESPONSE: l) Varies, but usually k to S members 
'i. WHAT PROBLEMS DIO YOU EXPERIENCE IN ESTABLISHING THE UNIT? 
RESPONSE: I) None 

5. WHAT SOLUTIONS WERE REACHED? 
RESPONSE: None 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT YOUR 
EVALUATION PLAN? 

RESPONSE: 0 None 
Additional Comments 

1) Old not like politics vjhich evolved with project and department. Granted 
there are politics within a university setting, and if you are going to 
play politics, then you have to expect that you may get burned. 



Additional Comments (continued) 

2) With respect to what m/RL wanted, Butman asked this question, "Are 
we off base or do they not know what they are talking about?'^ She 
felt MWRL was not off base. She felt alot of what they were doing 
could be translated Into the way things go and are, however some of 

it looks like someone dremed it up from some application of a systems 
technology theory or from an idea of what It ought to look like with 
out being involved in what tt does look like* 

3) Butman does not feel consortium has enough base in the social sciences* 

In the same view as 2) above, tfutman feels technology (statistical 
analysis) was used too extensively; feels it is just one tool among 
many as a means to solving problems; tendency then for technology to 
out run thinking then it comes time to operatlonalize* 

5) On the whole, Butman feels consortium Is an excellent concept* 

6) Since there Is not alot of good solid training of evatuators going on 
around the country, Butman feels the HTP was an excellent medium for 
training individuals to become competent evatuators* 

7) It is a must for Federal Government to supply adequate money resources* 

8) The HTP was an excellent concept as a work center* 

9) NWRL is fairly isolated geographically and, therefore, needs a meOIum 
.to reach other areas of the country* 

lO) Because each agency has Its own problems and contracts to meet, it is 
often difficult to have a fluid exchange of Ideas and matei lals from 
other agencies* 

It) The whole process of evaluation and consortium agencies Is a process 
which needs time to develop properly and concurrently to provide 
adequate services. 
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CONSORTIUH INTERVIEW 
Dr. Marvin Gewtrtz 
Institute for Educational Development 

May, 1973 
Interviewer; Etill Nealy 



Section I 

1. WHAT WERE THE NEEDS OF YOUR ORGANIZATION (RELATE TO): 

A) PROGRAMS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE; 1) Organizing and operating an evaluation staff In 
terms of evaluation theory and skills 

2) Center as a possible source of personnel a$ our 
organization expanded 

3) Valued evaluation centers expert view of our^ 
operation by site visit teem 

Did not capitalize very much from site ytslts 

5) Center and pur organization never really Jelled; 
we were expecting a lot more from the Center 

tn terms of help, training packages and seminars 

6) Became overly Involved tn tn-house problems; 
therefore, resigned because of the time situation; 
did select another member of our staff to serve 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 
RESPONSE; I) See response 1) above 

3. HOW DID EVALUATION EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

RESPONSE; 1) Work load allows little time for a comprehensive 
evaluation 



Section II 

). WHAT NEEDS, PROBLEMS, AND OPFORTUNlTIES INFLUENCED YOUR DECISION 
TO BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: I) See Section 1 above 

a) Prestige of being a member of the group 

b) The fact that we were selected 

c) We were the only agency of our .type 
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Section II (Continued) 



2. WHAT PROBLEMS OtO YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT 
FROM YOUR AGENCY? 

RESPONSE: I) None* everyone was Very happy 

3. IN REVIEWING THE CRITERIA FOR SELECTION . UHilH CRITERIA WAS THE 
HOST DIFFICULT? WHY? 

RESPONSE: I) We were in a special category 

2) We are not part of a larger organization* eg* 
a university; we are an R £^ 0 organization 

3) Were able to maintain workloads 

h) State of flux within the organization (change 
of presidents) 



Section 111 



1. WHAT ADVANTAGES DID YOU EXPECT TO GAIN AS A MEMBER OF THE 
CONSORTIUM? 

RESPONSE: 1) We would upgrade our staff 

2) Get new views of evaluation ^ 

2. WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 
RESPONSE: I) Cooimunicatlon with leaders in the field 

3. WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT OF BELONGING TO 
THE CONSORTIUM? 



RESPONSE: 1) Undersupply of personnel 

2) Linkage through one Individual was not efficient 

3) Wanted more student Involvement 

k) Our cycle of activities did not coordinate with 
the university schedulej therefore* It was 
difficult to release people 



TO WHAT EXTENT WAS YOUR AGENCY Ufl/OLVED WITH OTHER MEMBERS OF THE 
CONSORTIUM? 

A) SHARING OF MATERIALS 

B) WORKING OF RESOLUTION TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: I) No exchange other than two staff members were in 
comnunicatlon with Jim Jacobs of Cincinnati 

2) Site visit team to Cincinnati 

3) Jim Jacobs worked with Gewirtz on context evaluation 
system 

k)' Butman and lEO/Center and lED/Cincinnati and 

lED; lED only provided the Center with materials — 
and reports; Mitch Brickel gave several seminars 
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Section HI (Continued) 

5. WERE THE SiTrvilit^ BY THE MTP STAFF OF VALUE TO YOUR ORGANIZATION? 
IN WHAT WAY? 

RESPONSE: I) Yes, our people appreciated the ^act that we had 
the Site visit; they spoke openly about problems; 
it improved internal communications 

. 6. OID YOU PARTICIPATE IN ANY SITE VISITS? 

RESPONSE: 1) Yes, a rewarding experience; had an opportunity 
to see first hand evaluation operation other 
than our own 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVEO WITH 
THE MTP'S EVALUATION UNIT? 

RESPONSE: t) Participated in site visit with evaluation unit 
2) Filled out SAES and other materials 

8. IN WHAT WAYS OID THE PRDB SERVE YOUR AGENCY? 

RESPONSE: I) We contributed reports and other information 

9. 010 YOU OR MEMBERS OF YOUR ORGANIZATION l«RK WITH THE PROB STAFF? 
RESPONSE: t) Shared information on paraprofessionals 

10, WERE YOU OR MEMBERS OF YOIM STAFF INVOLVED IN SHARED ACTIVITIES 
WITH THE MATERIALS DEVELOPMENT COHPONEHT OF THE MODEL TRAINING 
PROJECT? 

RESPONSE: 1) See question 9 above 



Section IV 

1. TO WHAT EXTENT WERE YOU AND YOUR AGENCY iNVOLVEO IN THE PLANNING 
PHASES OF THE MTP PROJECT? 

RESPONSE: I) Were admitted at the Dallas meeting, but had very 
I ittle involvement 

2. OID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 
RESPONSE: 1) Yes, in Dallas and Shanty Creek we voted 

3. WERE YOU SATISFIED WITH YOUR OPPORUUNlTY TO TAKE PART IN MTP 
DECISION-MAKING? 

RESPONSE: t) Yes, a democratic operation 
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Section V 

\. HOW »MNV INTERNSHIPS WERE AVA ILABLE IN VOUR ORGANIZATION FOR 
MTP STUDENTS? 

RESPONSE: I) Never specified a nunber 

Z, WHAT PROBLEMS DID VOU FACE IN ACQUIRING INTERNSHIPS? 

RESPONSE: 1) Were witting to get some If they were available 
(Problem of schedules) 

3. WHAT PROBLEMS DID YOU EXPERIENCE tN RELATIONSHIP TO THE A'^VISEMENT 
OF INTERNS? 

RESPONSE: 1) Not applicable 
k. WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 
RESPONSE: I) Not applicable 

Section VI 

1. WHAT VJIAS THE TOTAL OPERATING BUDGET OF THE EVAmTION UNIT AS 
OF DECEMBER 30. |972? 

RESPONSE: I) Excess of $1.DD0.000 

2. HOW MUCH OF THE HONEY VMS SOFT HOMEY? 
RESPONSE: 1) All 

Section VI 1 

1. DID YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? 
RESPONSE: I) No 

2. TO WHAT fXTENT WAS THE EVALUATION COHPONENT DEVELOPED? 
RESPONSE: I) We expanded 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR UNITY 
RESPONSE: 1) Not applicable 

WHAT PROBLEHS DID YOU EXPERIENCE IN ESTABLISHI^:G THE UNIT? 
RESPONSE: I) Not applicable 
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Section VH (Continued) 

5r UtWT SOLUTfONS WERE REACHED? " 
RESPONSE: I) Not applicable 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT 
YOUR EVALIMTION PLAN? 

RESPONSE: 1) Not applicable 
Additional Comn^ents 

I) I was sorry to see the Center dissolved. 
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CONSORTIUH INTERVIEW 
Or* Robert Hainnond 
Hontana State Department 

May 3, 1973 
Interviewer: William Beruttt 



Section I 

K WHAT WERE THE NEEDS OF YOUR ORGANIZATION (RELATE TO): 

A) PROGRAMS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE: I) To develop an evaluation unit for obtaining {n- 
^ formation as It pertains to Improving educational 

practices; also to develop strategies and plans 
which will concurrently deal with Improving 
'^educa 1 1 ona I prac 1 1 ces 

2) Did develop a handbook for the state department 

a) Handbook laid out strategies for evaluation 

b) Developed fllmstrlps which were to be 
disseminated along with handbook for further 
clarification 

c) Purpose was to communicate what evaluation 
was doing 

3) Conducted systematic study of how state and 
superintendents Interacted with office of evaH 
uation and for Improving educational practices 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 

RESPONSE: I) Time for development of adequate strategies; 

lacking available staff with competencies to aid 
In the development of strategies and models; 
money became an issue to provide adequate resources; 
State Department had a tendency to view university 
personnel as theoreticians rather than practitioners; 
State Department was not readily available to aid 
in time or personnel 

3. HOW DID EVALUATION EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

RESPONSE: I) Conducted systematic study to identify information 
and decision-^maklng points 
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V 

Section 11 

1. WHAT NEEDS, PROBLEMS, ANO OPPORTUNITIES INFLUENCEO YOUR 
OECISION TO BECOME A HEHBER OF THE CONSORTIUM? 

RESPONSE: I) Prior association with Or. Stufftebeam 

Influenced the decision; saw Is. as an opportunity 
to get out Into the educational arena as a practi- 
tioner; also, opportunity to provide Information 
and feedback for the development of evaluation 
units 

2. WHAT PROBLEMS 010 YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT 
FROM YOUR AGENCY? 

RESPONSE r I) No problems 

3* IN REVIEWING THE CRITERIA FOR SELECTION WHICH CRITERIA WAS THE 
MOST 0 IFF I CULT? WHY? 

RESPONSE: I) The money NIE provided ytas tight* thereby* 
fostering limited resource allocations for 
personnel, Interns, and imptenientation; if NtE 
wants to experiment, they must provide adequate 
funding. 

2) Implementation of procedures as described by 
training institution (OSU) did not meet needs 
of State Oepartment; difficult getting practi- 
tioners to accept theory based approach 



Section 111 

K WHAT AOVANTAGES 010 YOU EXPECT TO GAIN AS A HEHBER OF THE 
CONSORTIUM? , 

RESPONSE: I) Sharing of Ideas, problems* interns, Imptemen* 

tation (very critical exchange aid In Institution*- 
allzatlon of evaluation) 

2. WHAT WERE THE AOVANTAGES ^ BELONGING TO THE CONSORTIUM? 

RESPONSE: I) Interaction with other consortium members, plus 

opportunity to provide personnel who are schooled 
for problems and evaluation within the state 
{to provide competent evaluators) 

3. WHAT PROBLEMS 010 YOU EXPERIENCE As A RESULT OF BELONGING TO 
THE CONSORTIUM? 

RESPONSE: 1) Problems resided basically within the administration 
of the state; this was not major^ yet could foresee 
It building on the horizon 
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Section Ml (Continued) 

't. TO WHAT EXTErrr VMS YOUR AGENCY INVOLVED WITH OTHER HEH8ERS OF 
THE CONSORT lUH? 
A) SHARING OF MATERIALS 

8) WORKING OUT RESOLUTIONS TO COHHON PR08LEHS 
C) EXCHANGE OF PERSONNEL 

RESPONSE: I) The extent of involvement with other agencies 
was very limited; Kanmond was involved In the 
beginning development of consortiun agencies; 
his consortium existed for onl/ one year before 
he left the Hontana State Oepar^nt and for the 
most part this dissolved any Interaction betwe^ 
Hontana and other consortiun agencies Including 
Ohio State University; a lot of early development 
consisted of training personnel and getting 
strategies In motion thereby ilmiting Interaction. 

5. WERE THE SITE VISITS BY THE HTP STAFF OF VALUE TO YORR ORGANIZATION? 
IN WHAT WAY? 

RESPONSE: 1) There were none 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 
RESPONSE: 0 No 

7. TO WHAT EXTENT WERE YOU OR HEH8ERS OF YOUR STAFF INVOLVED WITH 
THE MTP'S EVALUATION UNIT? 

RESPONSE: t) None 

8. IN WHAT UAYS DID THE PRD8 SERVE YOUR AGENCY? 
RESPONSE: l).--None 

9. DID YOU OR HEH8ERS OF YOUR ORGANIZATION WORK WITH THE PRD8 STAFF? 
RESPONSE: I) No 

f 

10. WERE YOU OP HEH&ERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES 
WITH THE MATERIALS DEVELOPHEttT COHPONEirT OF THE HODEL TRAINING 
PROJECT? 

RESPONSE: 0 No 



Section IV 

K TO WHAT EXTEttT WERE YOU AND YOUR AGENCY INVOLVED 1M THE PLANNING 
PHASES OF THE MTP PROJECT? (see next page) 
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Section tV CContinued) 

RESPONSE: I) Only Hammond was involved* through telephone 

conversations and group meetings at Ohio State 
University 

.2. DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 

RESPONSE: I) Yes 

3. WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART \H MTP 
OECISlON-MAKtNG? 

RESPONSE: 1) Yes 
Section V 

K HOW MANY INTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR 
MTP STUDENTS? 

RESPONSE: I) No response; did not reach stage of exchanging 
or receiving interns 

2. WHAT PROBLEMS DID YOU FACE iNACdUIRiNG SNTERNSHIPS? 
RESPONSE: I) Not applicable 

3. WHAT PROBLEMS DID YOU EXPERIENCE IN RELATIONSHIP TO THE 
ADVISEMENT OF INTERNS? 

RESPCWSE: I) Not applicable 

i*. WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 

RESPONSE: )) Not applicable 

Section VI 

1. WHAT WAS THE TOTAL OPERATING BUDGET CF THE EVALUATION UNIT AS 
OF DECEMBER 30, 1972? 

RESPONSE: I) $120,000- $150,000 

2. HOW MUCH OF THE MONEY WAS SOFT MONEY? 

RESPONSE: I) All; could not Set money from State, very limited 
local Support, ano 30% cutback as a result of new 
federal government cutbacks 
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Section VII 

1. OtO YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? (IF 
NO UNIT WAS INVOLVEO* GO TO NEXT SECTION) 

RESPONSE: 1) Yes, Title IV, k02 being Assistant Superintendent 
in charge of evaluation 

2. TO WHAT EXTEHT WAS THE EVALUATION COMPONENT OEVELOPEO? 

RESPONSE: I) Plan became operative and proceeded to being 
field tested within the State Department and 
within school districts; this Included gathering 
data, storage and retrieval of data, and compu- 
terization of data 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR UNIT? 
RESPONSE: I) Six 

WHAT PROBLEMS OID YOU EXPERIENCE IN ESTABLISHING THE UNIT? 

RESPONSE: t) Getting addttional funds to carry out activities 

2) Lack of understanding on part of State Department 
personnel for the need of evaluation unit to 
create and shape a plan to be implemented in 

V the State 

3) Any theoretical plan takes months maybe years 
to mold, shape, and functional i ze 

^) State Department did not like the way evaluation 
unit probed for information for development and 
dec'iston-making; tendency to have '*gut level'* 
dects ion^making 

S) State Department tended to respond negatively 
to the theoretical bases; did not realize the 
process involved to finalize models; plus the 
State Department was caught up in their own 
problems, thereby, providing little participation 
witiT^.evaluation un;t* 

5- WHAT SOLUTIONS WERE REACHED? 

RESPONSE: 1) Plan was put into action and provided information 
essential to declsloiv^making; however, often the 
Information was overlooked or was of controversial 
nature and was burled due to political entanglements 
2) System study Identified Information points and 

decision-making as they pertain to use and efflcienc 
of Title I and Ml; the process followed monies 
as It was disseminated from Superintendent down 
to the classroom; entered the classroom and worked 
with the teacher to study affect on the child; after 
studying the child and gathering Information, then 
followed process back to Superintendents through 

the evaluation unit 
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Section VI I (Continued) 

6. HOW HUGH INTERKAL TRAINING VMS NECESSARY IN ORDER TO IHPLEMENT 
YOUR EVALUATION PLAN? 

RESPONSE! I) First six months devoted to training plus much 
learning as plan was generated 



Additional Comments 

1. Consortium is an excellent concept and would like to see it 
expanded , 

2. tf a consortium is to develop sufficiently it must receive adequate 
support from the Federal Government. 

3' Consortium gives feedback to see what is needed in a university 
training setting. 

Since we do not have sufficient information on problems within 
school districts, the consortium can provide the linkage to the 
university to deal with such problems* 

5. Within the Montana State Department there was o lack of 
communication concerning intents, purposes, and understanding. 

6. Hammond's decision to leave concerned the State Department 
wanting him to sssume other administrative duties which did 
not pertain to his original responsibilities nor to evaluation* 
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CONSORTIUM INTERVIEW 
Dr* Phtlip Kearney 
Michigan Department of Education 

May 16, 1973 
Interviewer: Evelyn Brzezlnski 

This agency joined the MTP Consortium In late sunmer, 1972, and 

so was not actively involved In the Consortium for very long vl 
before the MTP*s demise. 'A 

Section I ■ "i 

1. WHAT WERE THE NEEDS OF YOUR ORGANIZATION (RELATE TO): i 

A) PROGRAMS 3 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE: 1) None . '! 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? j 

RESPONSE: I) None 1 

3- HOW DID EVALUATION EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

RESPONSE: I) None 



Section II 

K WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION 
TO BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: I) It was viewed as an opportunity to identify and 
recruit potential Department employees through 
the mechanism of the fnternshlp; it was also viewed 
as an opportunity for present Department employees 
to pursue graduate training In evaluation; Department 
personnel felt they could contribute to the MTP 
a *Vea|-wor|d*' experiential base from the point 
of view of a state agency 

2. WHAT PROBLEMS DID YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT 
FROM YOUR AGENCY? 

RESPONSE: I) There were no real problems, but it was a somewhat 
ticklish operation for a Michigan state agency 
to join an Ohlo^based consortium; since the 
Department personnel are quite non^parochlal , and 
because of Stuff lebeam's reco9nized expertise, 
however, there was not much of a problem 
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Section II (Continijed) 

3, IN REVIEWING THE CRITERIA FDR SELECTION , WHICH CRITERIA WERE THE 
MOST DIFFICULT? WHY? 

RESPONSE: )) Criteria ^f^ might have been a problem. . .Kearney 
was unsure as to whether he could commit as much 
time to the MTP as might be required; Criteria 
#7 likewise might have been a problem* . .It is 
unlikely that the Department could have paid 
direct tuition costs. . .it probably would have 
been covered by the salary paid to trainees 



Section 111 

K WHAT ADVANTAGES DID YOU EXPECT TD GAIN AS A MEMBER OF THE CONSORTIUM? 



RESPDNSE: 1) Those opportunities listed In Section ll» question 1 
above were two advantages; also* the consortium 
was viewed as a good sounding board for Ideas 
from Department personnel; finally, It was viewed 
as an. advantage froni a public relations point of 
view to have other Michigan districts see the 
Department and Saginaw working together in an 
evaluation effort 



2. WHAT WERE THE ADVANTAGES DF BELONGING TO THE CONSORTIUM? 

RESPONSE: )) Were not really a consortium member long enough 
to realize many advantages; some of the PR value 
mentioned above was attained, however 

3. WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT OF BELONGING TO 
THE CONSORTIUM? 

RESPDNSE: I) Not a member long enough to experience problems 

TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF 
THE CDNSORTiUM? 

A) SHARING OF MATERIALS 

B) WDRKING OF RESDLUTION TO COMMDN PROBLEMS 

C) EXCHANGE DF PERSDNNEL 

RESPONSE: I) There was quite a bit of involvement with 
Saginaw; Jerry Baker, from Saginaw, had a 
short-term internship at the Department; with 
other consortium agencies, however, there was 
no direct Involvement 



5, WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR DRGANlZATfON? 



RESPONSE: I) Were not visited 
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Section 111 (Continued) 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 
RESPONSE: 1) NO 

7. TO WHAT EXTENT WERE VOU OR MEMBERS OF YOUR STAFF INVOLVED 
WITH THE MTP'S EVALUATION UNIT? 

RESPONSE; 1) Nothing more than a few conversations with the 

people In that unit; involved to a minimal degree 
In the SAES Project (as a member of the Comnisslon 
for the Study of the Evaluation Profession) 

8. IN WHAT WAYS 010 THE PROB SERVE YOUt AGENCY? 

RESPONSE: 1) Not at all, but use undoubtedly would have 

been made of the pROB had the project continued 

9. 010 YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PROB STAFF? 
RESPONSE: t) No 

10. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES 
WITH THE MATERIALS DEVELOPMENT COMPONENT OF THE MTP? 

RESPONSE: 1) No 



Section IV 

1. TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED IN THE PLANNING 
PHASES OF THE MTP PROJECT? 

RESPONSE: 1) Not applicable due to late entrance Into MTP and 

short duration of agency's involvement in consortium 

2. DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 

RESPONSE: 1) Not applicable due to late entrance into MTP and 

short duration of agency's involvement In consortiimi 

3. WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE R(\RT IN MTP 
OECISION-MKING? 

RESPONSE: 1) Not applicable due to late entrance into MTP and 

short duration of agency's involvement In consortium 
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Section V 



HOW MANY lUTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR 
MTP STUDENTS? 



RESPONSE: I) One-three (or maybe more) would probably have 
been available had MTP continued 

2. WHAT PROBLEMS DID YOU FACE IN ACQUIRING INTERNSHIPS? 
RESPONSE: I) Not applicable 

3. WHAT PRD8LEMS DID YOU EXPERIENCE IN RELATIONSHIP TO THE 
ADVISEMENT OF INTERNS? 

RESPONSE: I) Not applicable 
^. WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 
RESPONSE: I) Not applicable 

Section VI 

K WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION UNIT AS 
OF DECEMBER 30, 1972? 

RESPONSE: 1) For the Research, Evaluation Assessment service 
area, the budget was $1, 024,000 

2. HOW MUCH OF THE MONEY WA^S SOFT MONEY? 

RESPONSE: I) Of $1,024,000, about $600,000 was from state 
appropriations; does not include Title I and 
Title III money in the "soft money" category 

Section VII 

K DID YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? 

RESPONSE: 1) There has been one since July, 1968; It has grown 
from a one-man operation to a thirty plus employee 
uni t 

2. TO WHAT EXTENT WAS THE EVALUATION COMPONENT DEVELOPED? 
RESPONSE: I) Not applicable 

3. HOW MANY PEOPLE WERE INVOLVEO IN YOUR UNIT? 
RESPONSE: I) Not applicable 
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Section Vn (Continued ) 

't. WHAT PROBLEMS DID YOU EXPERIENCE IN ESTABLISHING THE UNIT? 
RESPONSE: I) Not dppi tcable 

5. WHAT SOLUTIONS WERE REACHED? 
RESPONSE: I) Not applicable 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT 
YOUR EVALUATION PLAN? 

RESPONSE: I) Not applicable 



62 



58 



CONSORTIUM INTERVIEW 
Or. UiMiam J. Gephart 
Phi Delta Kappa 

April 11, 1973 
Interviewer: Evelyn Brzezlnskl 



Section I 

K mKJ WERE THE NEEDS OF YOUR ORGANIZATION (RELATE TO): 
A)' PROGRAMS 
' B*) PROJECTS 
C) ORGANIZATIONAL (PLANS) 

RESPONSE: 1) There were no real needs, but Involvement. 

in the consortium was viewed as one way to 
increase the level of Importance attributed to 
research and systematic on-going evaluation 
throughout PDK 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 
RESPONSE: I) None 

3. HOW DID EVALUATION AFFECT DECISIONS WHICH RELATED TO THE 
RESOLUTION OF NEEDS AND PROBLEMS? 

RESPONSE: 1) Not applicable 



Section II 

t 

). WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION 
TO BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: I) Of major importance was Gephart's previous 

working relationship wltn Stuff lebeam on the 
PDK Study Committee on Evaluation; the PDK 
involvement was* initially* Gephart's Involvement; 
but POK as an agency was to become much more 
involved once the MTP had products available 
for dissemination; POK's already existing dlssem* 
ination network was to be put to use for HTP- 
related products and processes 

2. WHAT PROBLEMS DID YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT 
FROM YOUR AGENCY? 

RESPONSE: 1) No problem 
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Section M (Continued) 



3* IN REVIEWING THE CRITERIA FOR SELECTION . WHICH CRITERIA WERE THE 
MOST DIFFICULT? WHY? 

RESPONSE: I) POK was not considered a f teld-ba^ed training 
agency and so criteria were not applied 



Section 1 1 1 

K WKAT ADVANTAGES 0)0 YOU EXPECT TO GAIN AS A MEMBER OF THE 
CONSORTIUM? 

RESPONSE: I) See response to Section ), question I, above 

2, WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 

RESPONSE: 1) tt gave Gephart the opportunity to continue to 
push for on-going evaluation services for POK 
at the headquarters level; it also gave htm 
insight Into practical evaluation problems so 
he could better focus and structure POK activities 

3, WHAT PROBLEMS 010 YOU EXPERIENCE AS A RESULT OF BELONGING TO 
THE CONSORTIUM? 

RESPONSE: 1) Ho real problem, but sofne frustrations: 

a) Oec 15 ion-making team meetings were not long 
enough to get ideas talked through 

b) Gephart did not have enough time to Interact 
and be involves as much as he would have 
Hked With MTr staff and students 

c) Even with the increased emphasis from I E/NIE 
on transportability, it became increaslr^ly 
clear that the MTP was more than anything else 
a process and thus transportability was not 
going to be easy to achieve 

d) USOE/NIE seemed unwUI Ing to accept sidetracks 
in the development schedule; For example, 
PROB was to be reorganized to fit SAES; 

PROB went as far as It could, but then had 
to stop and wait for SAES results; Federal 
officials wanted everything to be continually 
on the move and progressing, and that just 
was not aHiays possible 

TO WHAT EXTENT WAS YOUR AGENCY INVOLVEO WITH OTHER MEMBERS OF 
THE CONSORTIUM? 

A) SHARING Of MATERIALS 

B) WORKING OF RESOLUTION TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 
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Section HI (Continued) 

RESPONSE: I) Not much involvement with other consorttun agencies 

5. WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR 0RCANI2ATI0N? 
IN WHAT WAY? _ 

RESPONSE: .0 POK was not visited 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 
RESPONSE: I) No 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH 
THE MTP'S EVALIWTION UNIT? 

RESPONSE: I) Just as tt related to dects lon-maktng team meetings 

8. IN WHAT WAYS DID THE PROB SERVE YOUR AGENCY? 
RESPONSt': I) See 9 below 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PROB STAFF? 

RESPONSE: I) Gephart worked quite closely with the PROB and 
provided help on its structure; he was able to 
draw upon his knowledge of POK's existing Infor* 
mation storage and retrieval system/service 

10. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES 
WITH THE MATERIALS DEVELOPMENT COMPONENT OF THE MODEL TRAINING 
PROJECT? 

RESPONSE: 1) Just in decision-making team meetings 



Section IV 

1. TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED IN THE PLANNING 
PHASES OF THE PROJECT? 

RESPONSE: I) To a considerable extent; many of the ideas for 

the HTP were discussed among the PDK Study Committee 
on Evaluation of which Stuff lebeam and Gephart were 
two miend>ers 

2. DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 

RESPONSE: 1) Gephart was In conversations about membership 
selection, but he is not sure that is the same 
thing as actual participation In selection 
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Section IV fContlnued) 

^' lf!JLy°" SATISFIED WITH VOUR OPPCMtTUNiTV TO TAKE PART IN HTP 
OECIStON-HftKING? in nrP 



RESPONSE: 1) 



Vesj but the declsloij-iraiting team members were not 
really independent agents; all decisions were 
viewed as tentative, until follow-up with cooper- 
ating agencies was accomplished; because of the 
grant mechanism, actua} decisions were viewed 
as Stufflebeam^s responsibility and consortium 
msmbers saw themselves In an advisory role; the 
decision-making team meetings were not viewed as 
a mere academic exercise, however 



■Section y 

'* ??!!5EN?sf "'"^'^ ORGANIZATION FOR 

RESPONSE: I) None 

^* sJnJsf"^^"^ ^^"^ '^^^^^'^ ACQUIRING INTERN- 

RESPONSE: I) Not applicable 

^* rmSsr "'^ ^PERlEhCE IN REWTION TO THE ADVISEMENT 
RESPONSE: I) Not applicable 

^. WERE VOU SATISFIED WITH THE INTERNS VOU RECEIVED FROM THE MTP7 
RESPONSE: 1) Not applicable 



Secrtion VI 

t. WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION UNIT Ac 
OF OECEMBER, 19727 tVALUftllwi UNIT AS 

RESPONSE: I) There is no evaluation unit per se, but $50,000 
was allocated for the first time this year to 
use for Institutional research and evaluation 

2. HOW MUCH OF THE MONEY WAS SOFT MONEY? 

RESPONSE: 1) None 
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Section VII 

1. DID YOUR 0RCAHI2ATI0H INSTALL AN EVALUATION COMKJHEHT? 
RESPONSE: I) No 

2. TO WHAT EXTENT WAS THE EVALUATfOH COMPOHEWT DEVELOPED? 
RESPONSE; 1) Not applicable 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR UNIT? 
RESPONSE: I) Not applicable 

WHAT PROBLEMS DID YOU EXFcRIENCE IN ESTA6' ISHING THE UNIT? 
RESPONSE: ^) Not applicable 

5. WHAT SOLUTIONS WERE REACHED? 
RESPONSE: I) Not appitcable 

6. HOW MUCH INTCRKAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT 
YOUR EVALUATION PLAN? 

RESPONSE: I) Not appi icable 
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CONSORTtUH (NTERVtEW 
Or. James Jacobs 
Cincinnati 

April 2, 1973 
Interviewers: Ulttlani Berutti 
John Hllderbrand 
BID Nealy 



Section t 

1. WHAT WERE THE NEEDS OF YOUR ORGANtZATtON (RELATE TO): 

A) PROGRAMS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE: 0 Relationship with the consorttun was based 

on friendship, prestige, and status; t^e organiza- 
tion did not have any specific-needs; Cincinnati 
was concerned with what they woutd gain versus 
what received 

Z. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CtTED ABOVE? 

RESPONSE: t) Not applicable 

3. HOtf DID EVALUATION EFFECT DECtStOKS WHICH RELATED TO THE RESOLUTtDK 
OF NEEDS AND PROBLEMS? 

RESPONSE: i) Not app;:cable 



Section II 

1. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION 
TO BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: I) No problems; relationship based on trust between 
Adjunct Professor and Supervisor of project 

2. WHAT PROBLEMS Oil YOU EXPERIENCE IN GETTING A LETTER OF COHMJTMENT 
FROM YOUR AGENCY? 

RESPONSE: I ) None 

3. IN REVIEWING THE CRITERIA FOR SELECTION WHICH CRITERIA WAS THE 
MOST DIFFICULT? WHY? 
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Section M jContinued) 

RESPONSE: 0 Biggest problem was internship 

a) Had to modify contract. * .strike out the 
work training insert service 

b) First year consortium paid tuition the second 
year that was changed to a negotiation with 
each agency 

Section 1 1 1 

I* WHAT APVANTAGES DID YOU EXPECT TO GAIN AS A MEMBER OF THE CONSORTIUM? 

RESPONSE: )) See sections I and H above 
2. WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 

RESPONSE: I) See sections^ I and 1 1 above ^^S^ 

3* WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT OF BELONGING TO THE 
CONSORTIUM? 

RESPONSE: I) See section I and M above 

TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF 
THE CONSORTIUM? 

A) SHARING OF KATERIALS 

B) WORKING OF RESOLUTIONS TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: I) Before funding January to September^ 1971 attended 
two planning meet ings 

2) Large invoivement in selection of other consortium 
members 

3) Little involvement in budget matter; would like 
to have had more Involvement 

^) A conflict in conceptualization of training sites 
and training at the university 

5- WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR ORGANIZATION? 
I N WHAT WAY? 

RESPONSE: I) The site visits were valuable because of interaction 
with members of the MTP staff^ and the visists 
became a catalyst to move to solutions on problems 

6, 010 YOU PARTICIPATE !N ANY SITE VISITS? 

RESPONSE: 1) Cincinnati had one visit 
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Section III CConttnued) 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH 
THE MTP'S EVALUATION UNIT? 

RESPONSE: I) Very little 

8. IN WHAT WAYS DID THE PRDB SEkVE YOUR AGENCY? 
RESPONSE: I) Very little 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PRDB STAFF? 

RESPONSE; I) See question J and 3 above 

10. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SWVRED ACTIVITIES 
WITH THE MATERIALS DEVELOPMENT COMPONENT OF THE MODEL TRAINING 
PROJECT? 

RESPONSE; I) Materials shared at .quarterly meet ings 
Section IV 

1. TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVEO IN THE PLANNING 
PHASES OF THE MTP PROJECT? 

RESPONSE: I) Cincinnati was involved and contributed in the 
initial planning phase of the MTP 

2. DID YOU WRTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 

RESPONSE: I) Cincinnati acted as others did In an advisory 

capacity; the actual selections were left up to 
the project director and project supervisor 

3. WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IN MTP 
DECISION-MAKING? 

RESPONSE; 1) Except for budgetary decisions, was satisfied 

vjith their role In the planning phase of the MTP 

Section V 

I. HOW MANY INTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR 
MTP STUDENTS? 

RESPONSE: I) Two 
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Section V (Continued) 

2, WHAT PROBLEMS 010 YOU FACE IN ACQUIRING INTERNSHIPS? 

RESPONSE: 1) In order to,have interns, first must have an 

opening; second, wording had to be changed from 
"training" to "service" 

3. WHAT PROBLEMS OlD YOU EXPERIENCE IN RELATIONSHIP TO THE ADVISEMENT 
OF INTERNS? 

RESPONSE: i) Advisement of interns: 

a) Handbook (student and adjunct) helpful 

b) The internship coordinator was of great help 

WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 
RESPONSE: I) Yes, Cincinnati was satisfied with interns 

Section VI 

K WHAT WAS THE TOTAL OPERATING BUOGET OF THE EVALUATION UNIT AS 
OF DECEMBER, 1972? 

RESPONSE: I) Unable to specify adequately 
2, HOW MUCH OF THE HONEY WAS SOFT HONEY? 
RESPONSE: I) 95% soft money (estimate) 

Section Vn 

1- 010 YOUR ORGANIZATION INSTALL AN EVALUATiON CftiPONENT? 

RESPONSE: !) The Evaluation Unit was estabUshed prior to the MTP 

2. TO WHAT EXTENT WAS THE EVALUATION COMPONENT OEVELOPEO? 
RESPONSE: I) Not applicable 

3, HOW MANY PEOPLE WERE INVOLVEO IN YOUR UNIT? 
RESPONSE: I) Not applicable 

^> WHAT PROBLEMS 010 YOU EXPERIENCE IN ESTABLISHING THE UNIT? 
RESPONSE: I) Not applicable 
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Section VI I (Continued) 



5* W^tAT SOLUTIONS WERE REACHEO? 

RESPONSE: 1).. Not applicable 

6- HOW MUCH INTERNAL TRAINING WAS NECESSARY IN OROER TO IMPLEMENT , 
YOUR EVALUATION PLAN? 

RESPONSE; I) Not applicable 



Add i t lonal Comments 

1) Cincinnati's role in conceptualization of the HTP» and being 
an active participant in the operational phases^ was invaluable 
in the development of a viable program to train evaluators for 
the field* Cincinnati identified many problems as well as good 
points which surround a project such as the HTP, 
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CONSORTIUM INTERVIEW 
Dr. Jerry Walker 
Center for Vocational & Technical Education 

April 20, 1973 
Interviewer; Evelyn Brzezlnskt 



Section I 

K WHAT WERE THE NEEOS OF YOUR ORGANIZATION (RELATE TO): 

A) PROGRAMS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 
RESPONSE: I) None 

2. WtttT PROBLEMS WERE ASSOCIATED WITH THE NEEOS CITED ABOVE? 
RESPONSE: 1) None 

3. HCW DID EVALUATION EFFECT DECISIONS WHICH RELATED TO THE 
RESOLUTtON OF NEEDS AND PROBLEMS? 

RESPONSE: 1) None 



Section t 

1. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION 
TO BECOME A MEMBER OF THE CONSORTIUMS 

RESPONSE: \) There were no real problems or needs; prior 

association with Stufflebeam was the key factor; 
becoming a consortium [Rcniber was viewed as a good 
opportunity to learn and to get linkages to other 
agencies 

2. WHAT PROBLEMS DID YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT 
FROM VOUR AGENCY? 

RESPONSE: I) The amount of money involved was certainly no 

financial enticement; had to decide If the laborious 
details of belonging (e.g., added paperwork) were 
worth stiil-to-be determined gains 

3. IN REVIEWING THE CRITERIA FOR SELECTION , WHICH CRITERIA WERE THE 
HOST DIFFICULT? WHY? 
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Section 1 1 (Continued) 

RESPONSE: t) Commitment to provide Internship opportunities 
(this was always rather ill-defined); the Voc* 
Tech Center was not sure of their legal and 
pracLica) ability to provide internships, e.g., 
regarding legality, the VOc-Tech Center was not 
funded as a training agency; regarding practicality, 
other RA*s at the Center did not receive academic 
credit whereas MTP interns were to receive credit. * 
this could have caused problems 

2) Possible problem with the expectation that the 
evaluation capability of the Cenler would meet a 
threshold level as a result of MTP Involvement*** 
MTP was coincidental to growth, not consequential 
to it 

3) How to use the money received from the MTP was 
never adequately explained 



Lection HI 

K WHAT ADVANTAGES DID YOU EXPECT TO GAIN AS A MEMBER OF THE CONSORTIUM? 

RESPONSE: !) Learn more about how to do evaluation in an R&D 
setting 

2y Make contacts and meet liaison persons 
3) Extend association with Stufflebeam and other 
consortium members 

2. WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 
RESPONSE: I) Same as I above 

3. WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT OF BELONGING TO 
THE CONSORTIUM? 

RESPONSE: )) How to use the money provided by the MTP 

2) Dates for dectsion-making team meetings were 
not cleared in advance with adjunct professors 
(or at least not with Walker), and four or five 
times those meetings were held at inconvenient 

t imes 

3) Incessant questioning for information by different 
MTP staff and students (mainly in the form, of 
questionnaires, many of which were very time 
consuming). . .it seemed that a great deal of 

the information requests were redundant, as if 
one person in Oxtey didn't know what another 
one In the same building was doing 
't) Authority of the decision-making team was never 
reaily understood. . ^.there was never a really 
aJequate policy structure 
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Section i 1 1 (Continued) 

5) Consortium members always looked like 10-12 
Independent agencies; they never seemed to 
really cl Ick as a urbit, id Walker's opinion 

6) There did not seem to be much of a sense of 
purpose or direction from the MTP staff; it was 
frustrating to spend the required amount of 
time on the project {e,g<, filling out ques- 
tionnaires), without knowing why 

TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF 
THE CONSORTIUM? 

A) *^ SHARING OF MATERIALS 

B) WORKING OF RESOLUTION TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: I) Outside decision-making meetings, Walker 

received four^six contacts from other consortium 
members and initiated two~three more; contacts 
were mainly for k.S above, but some ^,A,;'the 
only kS. was that one of the Center^s MTP 
interns was from nWRL 



5. WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR 
ORGANIZATION? 



RESPONSE; I) They were not visited 
6. DID YOU PARTICIPATE !N ANY SITE VISITS? 



RESPONSE; I) No 



7- TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH 
MTP*S EVALUATION UNIT? 

RESPONSE: 1) Walker did not really identify evaluation as a 
distinguishable unit. He talked with Mary Anne 
Bunda about some problems on two or chree occasions; 
this informal consulting amounted to about eight 
hours 



8. IN WHAT WAYS OID THE PRDB SERVE YOUR AGENCY? 

RESPONSE: i) Not at all; once or twice Walker requested 
information on how to use the PRDB, but 
instructions were never given adequately 

9* OID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE pRDB STAFF? 



RESPONSE: D No 
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Section 1 1 1 (Continued) 

10, WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES 
WITH THE MATERIALS DEVELOPMENT COMPONENT OF THE MTP? 

RESPONSE: I) Walker met with Jack Sanders on several occasions; 

he viewed and critiqued two media-assisted presen-* 
tations prepared by the materials development unit 



Section IV 

1, TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED !N THE PLANNING - 
PHASES OF THE PROJECT? 

RESPONSE: I) Walker was quite heavily involved through informal 
discussion of ideas, plans, etc, with Stufflebeam; 
the agency was not involved ^ 

2, DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 
RESPONSE: I) No 

3, WERE VOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IN MTP 
DECISICN-HAKING? 

RESPONSE: I) Not really; there were three main problems: 

d) Ualker was not the official representative 
for about the first year of the MTP, , , 
there were two others before Walker, , . 
the transition from one person to another 
caused some problems 

b) The aosence of adequate procedures and 
policy guiding the decision**makin9 team 
rpeetings (see Section Ml, question 3 above) 

c) Walker had the feeling that most project 
decisions were made outside the decision* 
making team meetings; the team meetings 
were good for other reasons, but not for 
actually making decisions to guide the MTP 



Section V 

L HOW MAY INTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR MTP 
STUDENTS? 

RESPONSE: 1) Of 60-/0 RAs in the Vocational Tech, Center, 

probably 10 of the positions could conceivably 
be filled by MTP interns. They had only two 
interns howeve*" 

7() 
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Section V (Continued) 

2, WHAT PROBLEMS DID YOU FACE IN ACQUIRING INTERNSHIPS? 
RESPONSE: )) No problems 

3. WHAT PROBLEMS DID YOU EXPERIENCE IN RELATIONSHIP TO THE ADVISEMENT 
OF INTERNS? 

RESPONSE: I) No problems, he didn't actually advise either of them 
^. WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 
RESPONSE: I) Very much so 

Section VI 

1. WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION UNIT AS OF 
DECEMBER, 1972? 

RESPONSE: I) $200,000 

2. HOW MUCH OF THE MONEY WAS SOFT MONEY? 
RESPONSE: I) All 

Section VI I 

1. DID YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? 

RESPONSE: I) There was an evaluation unit before the MTP existed^ 
but it has grown to divisional status since the 
project began 

2. TO WHAT EXTENT WAS THE EVALUATION COMPONENT DEVELOPED? 

RESPONSE: I) It has grown from two persons four years ago to 
twelve persons today 

3. HOW MAMY PEOPLE WERE INVOLVED IN THE EVALUATION COMPONENT? 

RESPONSE: 0 Twelve, Including professional staff, research 
associates/interns, secretaries, and technical 
assistants 

^. WHAT PROBLEMS DID YOU EXPERIENCE IN ESTABLISHING THE UNIT? 

RESPONSE: I) There were not enough resources (personnel or 
monetary) allocated for about the first three 
years the evaluation unit was in existence 
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Section vn (Continued) 

5. WHAT SOLUTIONS WERE REACHED? 

RESPONSE: I) About a year ago, when the Center was reorganized, the 
evaluation unit was made into a division and Walker 
was offered the position of division head. Before 
accepting that position, he ascertained that adequate 
resources for the division would be available. 

6. HOW MUCH INTERMAL TWMNING WAS NECESSARY IN ORDER TO IMPLEMENT 
THE agency's evaluation PLAN? 

RESPONSE: 1) It has been very much a staff development program. 

They have gone from no evaluation unit at alt to a 
position of a pretty good understanding of what 
evaluation Is on the part of around ISO persons 



Additional Comments 

K The MTP staff seemed like firefighters at times. There were 
elaborate organizational charts, but no apparent purpose behind 
them, no philosophical base. It seemed that people failed to 
follow the axiom: "form follows function.** There was lots of 
"form", but little definition of "function." 

2. To say simpiy that there was a communication problem is like 
saying that an airplane crash occurred because of a gravity 
problem, but that definitely was a problem with the MTP based 
at Oxiey and the consortium members. * 

3. There seemed to be an overextension of intents In the whole 
Evaluation Center, not just In the MTP* This compounded the 
problem of a tack of ptirpose noted !n ff] above. 
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CONSORTIUH INTERVIEW 
Or* Wt) I lam Wayson 
Faculty of Educational Development 

June 8, 1973 
Interviewer: John A. HUderbrand 



Section \ 

K WHAT WERE THE NEEDS OF YOUR ORGANIZATION (RELATE TO): 

A) PRDBLEHS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE: I) A major problem was^f tndlng a way to pull 
the total Educational Development Faculty 
together 

2) A major project was the Evaluation of the 
Faculty Program; also the project was housed 
at The Ohio State University 

2- WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 

RESPONSE: I) Lack of cwimon program for the Faculty to 

congregate around; the MTP was at first seen 
to be this common rallying point; within the 
Faculty there was a problem of not enough coeston 

3^ HOW DID EVALUATION EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS ANO PROBLEMS? 

RESPONSE; I) Faculty members helped develop the MTP proposal; 

the Faculty of Educational Development voted 
to accept the project; the selection of personnel 
holding rank In the College of Education was voted 
on by Faculty and students 



Section H 

K WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION 

TO BECOME A MEMBER OF THE CONSORTIUM? , 

\ 

RESPOMSE: I) Same as Section 1, question;! 

2. WHAT PROBLEMS OlD YOU EXPERIENCE IN GETTING? A LETTER OF COMHlTHENT 
FROM YOUR AGENCY? 

R/-SPONSE: I) None 
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Section 11 (Continued) 

3* IN REVIEWING THE CRITERIA FOR SELECT IPN WHICH CRITERIA WAS THE 
HOST DIFFICULT? WHY? 

RESPONSE; I) The perception held by the Faculty and the 

Evaluation Center as to the role of the Center 
was different 



Section I 1 1 

K WHAT AOVANTAGES OlD YOU EXPECT TO QAtN AS A MEMBER OF THE 
CONSORTIUM? 

RESPONSE: I) Gains: evaluation services, program improvement, 
and a foca) point for the Faculty 

2. WHAT WERE THE ADVANTAGES OF BELONGING TO THE CONSORTIUM? 

RESPONSE: )) Advantages; contact with other consortium agencies, 
especially with Adjunct Professors considered a 
part of Education Development Faculty; support 
for students 

3. WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT OF BELONGING TO 
THE CONSORTIUM? 

RESPONSE: )) Problems: never really considered a member 
(in name only); did not receive national 
recognition, Ohio State University and the 
Evaluation Center did, but not Educational 
Deveiopment 

2) Internal strife between project and Faculty 
3^ Stated goals of the project were not the same 
as the written goals 

TO WHAT EXTENT WAS VDUR AGENCY INVOLVED WITH OTHER MEMBERS 
OF THE CONSORTIUM? 

A) SHARING OF MATERIALS 

B) WORKING OF RESOLUTION TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: I) Adjunct Professors were sent all Faculty 

mailings; very littte communication between 
Educational Development and consortium 

5. WERE THE SITE VISITS BY THE MTP STAFF OF VALUE TO YOUR 
ORGANIZATION? IN WHAT VWY? 

RESPONSE; t) No site visit 
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Section III (Continued) 

6. 010 YOU PARTICIPATE IN ANY SITE VISITS?* 
RESPONSE: I) Nominal participation 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVEO WITH 
THE MTP'S EVALUhTION UNIT? 

RESPONSE: I) Nominal participation wi th SAES 

8. IN WHAT WAYS 010 THE PROB SERVE YOUR AGENCY? 

RESPONSE: 1) Copies of materials from PROB were sent on 

request; the PROB never reached its full potential ' 

9. t)lO YOU OR MEnBERS OF YOUR ORGANIZATION WORK WITH THE PROB STAFF? 
RESPONSE: I) NO 

10. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVEO IN SHAREO ACTIVITIES 
WITH THE MATERIALS OEVELOPMENT COMPONENT OF THt MOOEL TRAINING 
PROJECT? 

RESPONSE: I) Some participation with ERIC and RUPS 
Section IV 

1. TO WHAT EXTENT WERE YOU ANO YOUR AGENCY INVOLVEO IN THE PLANNING 
PHASES OF THE HTP PROJECT? 

RESPONSE: I) Faculty and students participated in developing 
alternative strategies for the project (Advocate 
Teams) 

2. 010 YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 
RESPONSE: I) No 

3. WERE YOU SATISFIEO WITH YOUR OPPORTUNITY TO TAKE PART IN HTP 
OECISION-HAKING? 

RESPONSE: I) No; this was partly based on a misconception 
of the goals of the project and the ro?e the 
Faculty was to play 

Section V 

I. HOW MANY INTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR MTP 
STUOENTS? 
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Section V jConttnued) 

RESPONSE: I) Not applicable because most students received 
their course work at Ohio State University In 
the Educational Oevelopcnent Faculty 

2. WHAT PROBLEMS DID YOU FACE IN AC(iUiRIN6 INTERNSHIPS? 
RESPONSE: I) Same as question I above 

3. WHAT PROBLEMS 010 YOU EXPERIENCE IN RELATIONSHIP TO THE AOVISEHENT » 
OF INTERNS? 

RESPONSE: 1) Same as question I above 
^. WERE YOU SATISFIEO WITH THE iNTERNS YOU RECEIVEO FROH THE MTP? 
RESPONSE: I) Same as question I above 

Section VI 

1. WHAT WAS THE TOTAL OPERATING BU06ET OF THE EVALUATION UNIT AS 
OF DECEMBER. |972? 

RESPONSE: 0 Not applicable 

2. HOW MUCH OF THE MONEY WAS SOFT MONEY? 
RESPONSE: I) Not applicable 

Sect ion V 1 1 

1. OJO YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? 
RESPONSE: I) Not applicable 

2. TO WHAT EXTENT WAS THE EVALUATION COMPONENT DEVELOPED? 
RESPONSE: I) Not appi icable 

3. HOW MANY PEOPLE WERE INVOLVED IN YOUR IWIT? 
RESPONSE: I) Not applicable 

WHAT PROBLEMS DID YOU EXPERIENCE IN ESTABLISHfNG THE UNIT? 
RESPONSE: |) Not applicable 
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Section Vll (Continued) 

5. WHAT SOLUTIONS WERE REACHED? 
RESPONSE: I) Not applicable 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT 
YOUR EVALUATION PLAN? 

RESPONSE: 1) Not applicable 
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CONSORTIUM INTERVIEW 
Dr. Marvin Dawson 
Council of Great City Schools 

May, 1973 
Interviewer: Sill Nealy 



Section I 

1. WHAT WERE THE NEEDS OF YOUR ORGANIZATION (R" J\TE TO) : 

A) PROGRAMS 

B) PROJECTS 

C) ORGANIZATIONAL (PLANS) 

RESPONSE: I) Opportunity to' get personnel from credible organization 
in evaluation 

2) Help to upgrade city schools evaluation personnel by 
using interns in the field 

3) Could provide a field training base 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 

RESPONSE: 1) Many of the schools evaluation personnel were make-shift 

3. HOW DID EVALUATION EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

RESPONSE: 1) Did not get plans implemented 

2) Coursework to upgrade staffs In the field 



Section II 

t. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION TO 
BECOME A MEMBER OF THE CONSORTIUM? 

RESPONSE: I) The Council sees itself as a catalysis between the Center 
and the field training base 

2. WHAT PROBLEMS DID YOU EXPERIENCE IN GETTING A LETTER OF COMMITMENT FROM 
YOUR AGENCY? 

RESPONSE: I) None 

3. IN REVIEWING THE CRITERIA FOR SELECTION, WHICH CRITERIA WAS THE MOST 
0 IFF I CULT? WHY? 

RESPONSE: 1) We were a unique organization 
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Sect ion I H 

1. WHAT ADVANTAGES OlO YOU EXPECT TO GAIN AS A MEMBER OF THE CONSORTIUM? 
RESPONSE; I) See Section t above 

2. WHAT WERE THE AOVANTAGES OF BELONGING TO THE CONSORTIUM? 

RESPONSE; I) An opportunity to resolve problems 

3. WHAT PROBLEMS DID YOU EXPERIENCE AS A RESULT 0> BELONGING TO THE 
CONSORTIUM? 

RESPONSE: 0 How money would be used 

^. TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF THE 
CONSORTIUM? 

A) SHARING OF MATERIALS 

B) WORKING OUT RESOLUTIONS TO COMMON PROBLEMS 

C) EXCHANGE OF PERSONNEL 

RESPONSE: I) Site Visit 

2) When we had joint meetings 

5. WERE THE SITE VISITS BY THE MTP STAFF DF VALUE TO YOUR ORGANIZATION? 
IN WHAT WAY? 

RESPONSE; 1) Yes, the evaluation site team did a great job of 

spotting weaknesses and problems; It stimulated some 
action 

6. DIO YOU PARTICIPATE IN ANY SITE VISIT? 

RESPONSE; 1) Yes, Saginaw Michigan Site Team 

1, TO WHAT EXTtNT WERE YOU OR tIEMSERS OF YOUR STAFF INVOLVED WITH THE MTP' 
EVALUATION UNIT? 

RESPONSE: 1) SAES and site visits 

8, IN WHAT WAYS DID THE PROB SERVE YOUR mGENCY? 
RESPONSE: I) Provided information 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PROB STAFF? 
RESPO. j£: 1) No 

to. WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES WITH 
THE fWTERIALS OEVELOPMEHT COMPONENT OF THE MODEL TRAINING PROJECT? 

RESPONSE: I ) No 
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Section IV 

1. TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED IN THE PLANNING PHASES 
OF THE PROJECT? 

RESPONSE; l) None 

2. DID YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 
RESPONSE: I) Yes 

3. WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IN MTP DECISION- 
MAKING? 

RESPONSE: t) None 
Section V 

1. HOW MANY INTERNSHIPS WERE AVAIUBLE IN YOUR ORGANIZATION FOR MTP STUDENTS? 
RESPONSE: Not Applicable 

2. WHAT PROBLEMS 010 YOU FACE IN ACaUlRlNG INTERNSHIPS? 
RESPONSE: Not appUcable 

3. WHAT PROBLEMS DID YOU EXPERIENCE IN RELATIONSHIP TO THE AOVISEMENT OF 
INTERNS? (EXAMPLE: INFORMATION CONCERNING PROBLEMS YOU EXPERIENCEO) 
IF ANSWER TO ONE WAS NO, GO TO NEXT SECTION. 

RESPONSE: Not applicable 

^. WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 

RESPONSE: Not AppUcable 

Section VI 

I. WHAT WAS THE TOTAL OPERATING BUDGET OF THE EVALUATION UN'T AS OF 

OECEMSER, 30, 1972? (IF NO BUDGET WAS AVAILABLE, GO TO NEXT SECTION) 

RESPONSE: I) We did not have an evaluation unit; we assisted schools 
in improving their evaluation capablltt'es 

Section VI I 

1. OID YOUR ORGANIZATION tNSTALL AN EVALUATION COMPONENT? (IF NO UNIT WAS 
ESTABLISHED GO TO NEXT SCCTION) 

RESPONSE: Not /.pplicable 

8*) 
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Section VI 1 (Continued ) 

2. TO WHAT EXTENT WAS THE EVALUATION COMPONENT uEVELOPEO? 
RESPONSE: Not Applicable 

3. HOW MANY PEOPLE WERE INVOLVEO IN YOUR UNIT? 
RESPONSE: Net Applicable 

WHAT PROBLEMS OlO YOU EXPERIENCE IN ESTABLISHING THE UNIT? 
RESPONSE: Not Applicable 

5. WHAT SOLUTIONS WERE REACHEO? 
RESPONSE: Not Applicable 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN OROER TO IMPLEMENT YOUR 
EVALUATION PLAN? 

RESPONSE: Not Applicable 

Add; t ional Conwaents 

Ideas of a consortium supported training program is a very good one. 
The U.S. Office of Education did not allow enough time to let the organization 
so I i d i f y . 
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SUHKARY AND CONCLUSIONS 

This section of the report provides the summary and conclitston 
of the responses to the questionnaire by section. This Is based 
on both the responses to the interview questionnaire and the authors 
perception of the HTP's situation. The preparation for this section 
required not only extensive reading of all or most of^ the HTP documents^ 
but a great deal of travel and time on part of present staff. ^ 

Section \ 

K WHAT WERE THE NEEDS OF YOUR ORGANIZATION (RELATE TO): 

(A) PROGRAMS 

(B) PROJECTS 

(C) ORGANIZATIONAL (PLANS) 

2. WHAT PROBLEMS WERE ASSOCIATED WITH THE NEEDS CITED ABOVE? 

3. HOW DIO EVALUATION EFFECT DECISIONS WHICH RELATED TO THE RESOLUTION 
OF NEEDS AND PROBLEMS? 

The first section of the interview questionnaire was designed to 
be a general overall picture of each consortium member prior to mem* 
bership tn the MTP^ Section I Is subdivided in three major areas: 
needs^ problems^ and effect. 

Most agencies expressed their needs in terms of I) recruiting^ 
2) training^ and 3) upgrading personnel from both within the organiza* 
tion and outside. There was a problem of recruiting qualified people 
to fill vacancies. The agencies viewed the MTP as a bank for trained 
personnel either through hiring one of its graduates or through 
the internship concept. The agencies expressed an interest In upgrading 
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evaluation within their own agency and also nationally, and through 
this upgrading process, materials, ideas, and people could be exchanged 
through the consortium network. 

There were problems associated with the consortium involvement 
from the outset of the project. Most agencies agree that money, 
obtaining interns^ released time for agency staff, providing credit 
for field based trdining, and tuition were cited as sdne of the pri- 
mary problems. 

Section ! I 

1. WHAT NEEDS, PROBLEMS, AND OPPORTUNITIES INFLUENCED YOUR DECISION 
TO BECOME A MEMBER OF THE CONSORTIUM? 

2. WHAT PROBLEMS 010 YOU EXPERIENCE IN GETTING A LETTER JF COMMITMENT 
FROM YOUR AGENCY? 

?. (N REVIEWING THE CRITERIA FOR SELECTION, WHICH CRITERIA WAS THE 
MOST DIFFICULT? WHY? 

Section II attempts to document why agencies became members of 
the MTP. What vjere the influencing factors?; what problems were 
associated w'lth committing agency resources?; and what criteria was 
the most difficult to comply with? 

Probably the reason most agencies consented to be'^ome members 
fn the MTP is because of prior association with Or. Stuff^ebeam. 
Becouse of his visibility and his recognition in the field of evaluation 
it was thought beneficial to have the agency associated with the project 

Also the agencies had the opportunity to exchange ideas across 
o»-her agencies while receiving either expertise or interns. 

Most agencies reported there was no problem associated with getting 
a (etter of commi tment . There were some minor concerns with this, 

8*) 
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such as, putting resources in an out-^of^state based project, or the 

wording of the contract. These concerns were experienced but were 

ironed out quickly. 

The criteria for selection most difficult to comply to as cited 

by the agencies were as follows: 

K The agency shall commit resources to implement the 
institutionalization plan. Support by the agency for 
staffing, for Installation of the evaluation unit, and 
for operation of the evaluation unit ts estimated at 
$200,000 per year or some reasonable amount of the 
operating budget. 

The agency shall ccmimit an on-staff person (Ph*0, level) 
as an adjunct professor of the Academic Faculty of the 
Educational Development Department at The Ohfo State 
University to direct the on-^site training in evaluation 
for trainees. This person shall commit himself to; 

a) conduct regularly scheduled seminars for trainees 
within the developed guidelines of the project, 

b) attend all meetings and workshops scheduled for agency 
representat ives 

S. The agency shall serve as a recruitment and placement 
agency for trainees of the program. 

7- The agency shall coavnit itself to pay support and to pay 
tuition costs for all trainees^ receiving Ohio State 
university credit in the time t^ey are based as an intern 
In that agency. 

And 5 and 7 were by far the most difficult. 



Section 1 1 1 

1, WHAT ADVANTAGES DID YOU EXPECT TO GAIN AS A MEMBER OF THE CONSORTIUM? 

2. WHAT WERE THE ADVANTAGES OF.. BELONGING TU THE CONSORTIUM? 

3- WHAT PROBLEMS D!D YOU EXPERIENCE AS A RESULT OF BELONGING TO THE 
CONSORTIUM? 

TO WHAT EXTENT WAS YOUR AGENCY INVOLVED WITH OTHER MEMBERS OF 
THE CONSORTIUM? 

(A) SHARING OF MATERIALS 

(B) WORKING OUT RESOLUTIONS TO COMMON PROBLEMS 

(C) EXCHANGE OF PERSONNEL 
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5 WERE THE SITE VISITS BY THE HTP STArF OF VALUE TO YOUR ORGANIZAT!ON? 
IN WHAT WAY? 

6. DID YOU PARTICIPATE IN ANY SITE VISITS? 

7. TO WHAT EXTENT WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED WITH 
THE HTP*S EVALUATION UNIT? 

IN WHAT WAYS 010 THE PROB SERVE YOUR AGENCY? 

9. DID YOU OR MEMBERS OF YOUR ORGANIZATION WORK WITH THE PROB STAFF? 

10, WERE YOU OR MEMBERS OF YOUR STAFF INVOLVED IN SHARED ACTIVITIES 
WITH THE MATERIALS DEVELOPMENT COMPONENT OF THE MODEL TRAINING 
PROJECT? 

Section III deals with inany issues involved in the participation 
of the consortium in the MTP. The following outlines the expected 
gains, actual advantages^ and the problems associated with membership 
in the MTP: 

1 , Expected gains 

a) receiving personnel from a credible organization 

b) contact with other agencies 

c) association with Dr, Stufflebeatn 

d) staff development 

e) visibility 

f) exchange of materials^ reports, etc, 

2, Actual advantages 

a) provided the opportunity to push for on**going evaluation 

b) all the expected gains listed in column I; the degree 
of satisfaction varies with each expected gain 

3, Problems - 

a) cycle of activities for placing intern sometimes 
out-of'phase with agencies 

b) need more than one contact person 

c) changing of out-of~state tuition 

d) no release time for agency staff 

e) need more student involvement 

f) incessant questionnaires by both staff and students 

g) authority of decision making team never really understood 

h) very Uttle exchange of personnel 

The problems listed above reflect the degree of satisfaction of the 
actual benefits in belonging to the HTP, 
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The Ihird year would have concluded many of the project activities. 
For example^ the PRDB would have been computerized, SAES would have 
been field tested and in operation^ and materials development would 
have finished and field tested some of their materials. Many indi^ 
viduals would' have graduated during the third year and follow-up 
could have been started in order to document their impact. 

Most of the agencies agreed that the components of the MTP 
had not matured prior to the ending of the project. Some of the 
adjunct professors Jid participate in site visits, and some, but not 
all agencies were visited. Some of the adjunct professors did 
particioate, both with SAES and the PRDB, but not to a great extent* 
The exchange of materials between agencies happened, but not at a 
rate much greater than would have happened normally. Those who did 
participate in MTP activities for the jnost part felt positive about 
the effort* All agencies did receive some material from the MTP* 
These materials were: Student Handbook , Adjunct Professor Handbook , 
and the MTP brochure. This was in addition to materials handed out 
t»t the quarterly consortium meetings. All agencies were visited 
by Otane Reinhard, Consortium Liaison Director* Her invoIve?iient 
in the capacity was seen as a definite plus by all agencies. 

Section IV 

U TO WHAT EXTENT WERE YOU AND YOUR AGENCY INVOLVED IN THE PLANNING 
PHASES OF THE MTP PROJECT? 

2. OfO YOU PARTICIPATE IN THE SELECTION OF MEMBERS OF THE CONSORTIUM? 

3- WERE YOU SATISFIED WITH YOUR OPPORTUNITY TO TAKE PART IN MTP 
DECISION MAKING? 
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Section IV attempts to document the consortium participation in 
the planning phase of the project, selection of members^ and HTP 
decision making. 

The consortium membership changed during the life of the project. 
When the project was originally started^ only a few participated in 
the plenning phase. However^ all consortium manbers were involved 
in the addition or deletion of member agencies* Some of the agencies 
felt their participation in the decision making was in an advisory 
capacity only. Some agencies felt that budgetary decisions^ expecially 
those decisions relating to interns and the consortium should be 
made by the consortium at quarterly Jecision making meetings. 

Section V 

L HOW MANY INTERNSHIPS WERE AVAILABLE IN YOUR ORGANIZATION FOR MTP 
STUDENTS? 

2, WHAT PROBLEMS DID YOU FACE IN ACQUIRING INTERNSHIPS? 

3. WHAT PRC3LEMS DID YOU EXPERIENCE IN REtATlONSHiP TO THE ADVISEMENT 

OF INTERNS? (EXAMPLE: INFORMATION CONCERNING PROBLEMS YOU EXPERlENCfO) 
IF ANSWER TO ONE WAS NO, GO TO NEXT SECTION. 

WERE YOU SATISFIED WITH THE INTERNS YOU RECEIVED FROM THE MTP? 

Section V deals specifically with the interns. It looks at the 
problems of acquiring interns, advisement of tnterns, and whether 
the agency was satisfied with the interns* 

This document deals only with interns in one or more consortium 
agencies. There w^re other internship positions such as working directly 
with a professor on a research related experience, or working within 
the Evaluation Center operating a separate project, and a third possibility 
was working in a nonconsortium agency. Host of the internships were 
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done in an agency that was a member of the consortium. Based on the 
interviews, the most internship positions available in an agency 
was six. The most positions ever filled by interns was three and 
least positions for internships was none. In most cases the position 
earmarked for Interns were cyclic in nature, fluctuating from none 
CO two or three. Those agencies that received interns stated they 
were satisfied with the quality of the work performed by the student* 

Host adjunct professors were not familiar enough with The Ohio 
.^tate University, especially specific courses and professors, to 
adequately advise. The MTP Student Handbook and the MTP Adjunct 
Professor Handbook were very useful* Host adjunct professors stated 
^he Consortium Liaison Coordinator was very helpful tn providing 
aid for advisement. 

Section Vt 

1. WHAT WAS THE TOTAL OPtRATING BUDGET DF THE EVALUATION UNIT AS OF 
DECEMBER 30, 1972? (IF NO BUDGET WAS AVAILABLE, GO TO NEXT SECTION) 

2. HOW HUCH OF THE MONEY WAS SOFT MONEY? 

Section Vi attempts to ascertain the amounts of money available 
for evaluation. Since evaluation cuts across many areas within any 
agency, the budgetary considerations also cut across many areas. 
Therefore, it was very difficult to obtain figures that would accurately 
describe the total picture. The highest amount allocated to evaluation 
according to the adjunct professors was $1,02^,000. The smallest 
amount was $50,000. The percentage of soft money ranged from 100% 
to 35X, 
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Section VI 1 

1. 0)0 YOUR ORGANIZATION INSTALL AN EVALUATION COMPONENT? (IF NO UNIT 
WAS ESTABLISHED GO TO NEXT SECTION) 

2. TO WHAT EXTENT WAS THE 'VALUATION COMPONENT DEVELOPED? 

3. HOW MANY PFOPLE WERE INVOLVED IN YOUR UNIT? 

WHAT i^ROBLEMS DIO YOU EXPERIENCE IN ESTABLlSHINf THE UNIT? 

5. WHAT SOLUTIONS WERE REACHED? 

6. HOW MUCH INTERNAL TRAINING WAS NECESSARY IN ORDER TO IMPLEMENT 
YOUR EVALUATION PLAN? 

Section VII deals with the installation of an evaluation unit. 
In most cases this is inappropriate because most of the consortium 
a9cncles had an existing evaluation unit. Some of the agencies that 
had evaluation units expanded considerably in size. One agency 
i^ent froni a staff of one to a staff of thirty anployees; another 
expanded from a staff of one to a staff of seven; and another expanded 
from a staff of two to a *>taff of twelve* This is not to say that 
the MTP was the cause for the expansion, but hopefuHy the MTP helped. 

The following "aJditional comments** were provided by the adjunct 
professors during the interview sessions. These comments are directed 
toward those attempting to build a project similar to the MTP, 

I, There should be two internships for each student. The 
first, lasting two or three days. Primarily for the 
purpose of visiting a cross sample of the agencies. 
This would enable the student to structure a better 
program while at the uni^'ersity. The second internship 
should last approximately six months. If the internship 
is funded by the consortium, an intern would receive ^ 
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variety of experiences. If the agency pays them, 

the intern would be assigned to a particular position. 

Avoid political confrontation that involves the effectiveness 

of the project. 

The consorti.um is an excellent concept and should be 
expanded. 

If the consortium is to develop Sufficiently it "^ust 
receive adequate Support frdn the Federal government. 
The consortium gWes feedback to see what is needed 

in a university training ..sessS-on, - ^ 

Since universities do not have sufficient information 
on problems within school districts, the consortium 
can provide the linkage to the university to deal with 
Such problems. 
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VfTA 

The Center for Research and Leadarship Oevatopment 
In Vocational and Technical Education 
The Ohio State University 

The organizational structure, program, and related resources oF 
The Center have a unique contribution ta make to the proposed project, 
"New Patterns for Training Research^ Development, Demonstration^ Dis- 
semination and Evaluation Personnel In Education/.' The Center is 
organized as an Independent unit on The Ohio State University campus and 
operated directly under the Office of the Provost and Vice President 
for Academic Affairs. It is mul t t-^disc ipla inary In its staff and orten-^ 
tation, fnter-institutfonai in its program and operat ional commltmentt» 
dnd coniprchensive in its focus on the full ronge of problems surrounding 
vocational and technical education. 

In addition to the research, development, diffusion^ and training 
components, it also operates an ERIC Clearinghouse as an integral part of 
the orgainzation structure. The mission of The Center is: 

To strengthen the capacity of state educational systems to provide effec- 
tive occupational education programs consistent with individual needs and 
manpower requirements, through 

(a) Research and development to fill voids in existing knowledge 
and to create, methods for*applying (cnowtedge. 

(b) Programmatic focus on state leadership development, vocational 
teocher education, curriculum development strategies, vocational choice and 
adjustment. 

(c) Stimulating and strengthening the capacity of other agencies 
and institutions to create durable solutions to significant problems. 

(d) A national information storage, retrieval^ and dissemination 
system for vocational and technical 'education throughthe affiliated ERIC 
Clear i ng^ouse. 

The Center is organized to optimize programmatic research and 
development, focusing on significant problems in vocational and technical 
education. Six major program areas are maintained* several of these 
directly relevant to the proposed project. Further, the staff of The 
Center provides research, development, diffusion, and evaluation exper- 
tise. The ERIC Clearinghouse provides an added dimension In information 
systems and information analysis. The senior staff of The Center hold 
academic t:tles in rf^levant departments of The University. At the present 
time, eighteen hold academic appointments from seven different departments 
in four different colleges or schools. 
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During the past several years. The Center has •'»9^S«d in a wide 
range of activities which substantively end methodological y ^^'t'^^^'V 
to the proposed project. It has also had extensive rPri'"^' 
blishleg, operating and participating In consortia. To '"^^ 
tive relationships have been established and maintained with seven 
fe5Sional associations, forty-one universities and coUegas^ three community 
colleges, and fifteen sUte departments of education. 
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VITA 

Ctnclnnatl^ Ohio PubUc School System 

The Cinctnnat) Public Schoot Systm servei 85,000 children In lOB 
buildings. The central administration, headed by the superintendent, 
Is divided Into five departments: EducaCtonet Program; Field Hanagament; 
Business; Student Development and School Plent* The first two departments 
are heeded by Associate Superintendents, the latter three by Assistant 
Superintendents. 

The Oepertment of Educettonel Program consists of four divisions: 
Instructional Services; Program Research and Design; Cveluatlon Services 
end Resource Services. Instructional Services contelns some forty sub- 
ject matter and curriculum specialists who develop curriculum, super* 
vise classroom teachers, and provide In-servtce trelning programs to taach- 
ing staff. One of the major tasks of these specie} Ists Is the production 
of curriculum bulletins (courses of study). This grOupttso^ has major 
responsibit Ity for Input evaluation and pilot Implementatfon of Innovative 
curricula, e*g. PSC, BSCS, SMSE, Suttlvan Reading, etc. All persons In this 
division are certificated, with Haster^s Degrees, usually In subject areas. 
One (necnber of the staff holds the doctorete. 

The Division of Program Research and Design serves as a catalytic 
agent for Instructional innovation thus working closely with Instruction- 
al Services* The staff consists of nineteen full time professional people 
engaged in research, development, and evaluation activities* Six evaluation 
specialists are charged with Title I, Title III, and state compensating 
program evaluation* Jm researchers are developing a school Information 
system aided by a proorammar analyst and statistician (Title IK)* A 
project coordinator and six teachers are developing a three»year*old pre-* 
school program (Title HI). Finally, two parsons are responsible for pro- 
gram design, mainly of federal and state program* The division Is headed by 
a director and one associate director for Research and one associate 
director for Design (development)* Of the staff of nineteen, six hold 
doctoral degrees while two others will complete their degrees this year. 
This division sulmlts many grant applications and manages several distinct 
projects. It is administratively responsible for one elementary school 
designated as an R&D Center. Research by outside persons and agencies Is c 
channeled through this division which also produces the Jo urnal ^ of P^rogram 
Research and Development (about twenty-ffve volumes over*7He past ffve yeers). 
In 1966 the Office of Education published 100,000 copies of the TIt;e I ' ' 
evaluation report of this division for national distribution. 

The Division of Evaluation Services has ni^t full time people, 
all with H.A.'s. Five of these people serve as group test examiners, ad« 
ministering such tests as achievement, I.Q., G.E.E., Advanced Placement, 
5.A,T., etc* Thus, the division administers the school testing program. 
In addition, the remaining three persons conduct systemwlde studies on such 
niatters as profit mobility, and follow-up studies* At present they are 
developing a pupil sampling IttwiL sampling technique (with £TS) to measure 
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alt forms of student achievement !n the ninth grade. 

The Division of Resource Services Is a central library for audio- 
visual aid5* It evaluates and 5elects A*V (naterials which are sent to 
schools upon request. The staff has four professional persons including 
a specialist in multimedia and a resource center supervisor* 

These four divisions comprising the Department of Educational Program 
would be most directly involved in the consortium. 
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VITA 

The Council of the Great City Schools 

The Council of the Great Ctty Schools Is a ccnsorttum of twenty*-c»>') 
of the largest school districts In the United States* It represents the 
large cities, and the convnon thread which holds the organization togethe'^ 
Is urban education and the problems faced in the urban setting* Within 
the member districts of the Council are some five million public school 
pupils, or approximately 12% of the national public school population* 
The Council ts the only national organization whose total concern Is 
urban education, and It Is the only one which has both school superln^ 
tendents and Board of Education members on Its own Board* 

The Council has three major priority areas* Under the first general 
umbrella Is equel educational opportunity, and there ere now projects 
funded tn this area for technical support with problems of desegregation, 
for training of minority group representatives for school administrative 
positions^ and for exploration of student participation In the decision- 
making process In school operation* The second general area Is currlculun^ 
and the major effort here Is a large developmental project In vocational 
education, which Is currently In the planning stage* 

The third major area of Council Interest Is In Improved management 
practices In large urban school districts and In supportive research and 
evaluation activities* The largest project In this area Involves the 
development of a model management Information system In the Dallas Inde- 
pendent School District* The Dallas HIS Is being built as an extension 
of the Belmont Information System of the Office of Education* it Is 
meant to be adaptable toother large city school systems, and It Involves 
extensive training and orientation among the Councils other twenty members. 
The Council also has management support contracts In the performance con- 
tracting area within three of Its member districts* 

Of more direct Interest, the Council has been engaged In research 
training as part of Its support of the general management area for 
some three years* Two research training projects are currently funded 
by the Office of Education and are administered directly by Council staff. 

The first of these Is an In-^servlce project which has provided a 
series of three-^day sessions for research directors and for members of 
research division staffs. Such subjects as general measurement theory 
evaluation, proposal writing. Instructional systems, regression analysis, 
attitude measurement, and systems approaches to the solution of educa- 
tional problems have been included. This series has been quite well re* 
celved within member districts and Is generally cited as one of the 
Council's most useful activities. 
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The second training project Is largely presevtce-oricnted and ts In 
three parts. Four interns are located In the Council s central offices 
in Washington, O.C* and have been Involved {n the special areas of tast{n9» 
performance contracting, research publications, and curriculum development 
in vocational education. Under subcontract to the Council the School 
District of Philadelphia is training six interns. These young people 
work half*tim^ in the Office of Research and Evaluation of the Phttadelphta 
Schools and take format courses at the Univeritty of Pennsylvania the other 
half of their time. They will receive H4A. de9raas tn educational research 
at the end of two years providing they meet all requirements of tha Univer* 
stty. The third part of the program Involves development of e evalua* 
tton model and related triinlng acttvtties at Teech«rs' College^ Colon- 
bra University, This ts handled on a purchase^oNserviceS arrengerMnt. 

The Council thus has experience in research training^ and this con* 
tinues to be an area of spectel interest* ft should be underftood* of 
course, that "reseerch** is being used int its broadest sense In the 
CouncI I 's activities* 

The Council brings to the Consortium its own personnel resources* 
Far more than that, however^ It can draw upon the vaster resources of the 
twenty-*one districts which ccQprise its membership* It Is the hub of a 
communications network among these districts, and it can reach Specific 
dectsion^makers and affect decisions mure quickly and mora effectively than 
is possible in any other way. On many important issues^ when it has been 
empowered by its Board of Directors^ the Council speaks for its members* 
In a real senle the Council of the Great City Schools represents the major 
urban school districts and their interests, and it provides the mechanism 
through which 'nany problems of urban education m&^^iSQH, f^' ^t^^ 
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VtTA 

Dall^jis Int::>DQndent__Schso] District 

The following doscription is on attempt to highUght those institu" 
ttonal, progrotnmotic, ond staff characteristics of the Oallos Independent 
School District whtch scent to be most immediotely and beneflctolly related 
to its role as o Field Training Complex !n a Model Training Program for 
RDD&E personnel* 

In^titutipnal Structure for RDD&£ 

The main locus for RDD&E activities tn the District Is the Division 
of Development, ijhich Is one of the three divisions tn the District, eoch 
being heoded by on Associate Superintendent who onswcrs directly to the 
Generol Superintendent. The Development Division Is divided Into two 
broad areos'^'lnnovotWe Leorning Systems Projects and the Deportment of 
Plonnlng, Research, and Evaluation, which department Is divided Into 
three functional groups, os follows: 

1 ♦ Information Svst^s Group flSC) 

--data processing and analysis Interface between user departments and 
the regional canputer utility. Planning and Management Information System 
(R'llS) development ond operotlon, and computer applications design* Prime 
locus of most context evaluation of an Institutional research nature* 

2* PlaflnInQ S:gtems Group (PSG ) 

"long-range planning, operational planning, developmont and reflne*- 
ment of PI-IIS and Interface with PPBS* Assists tSG In context evaluation 
anc* PMtS* Prime locus of | np M^ evaluation * 

3# Research and Evaluation Branch tREB) 

"resource for all research and evaluation design, program auditing, 
and program assessment* Prime locus of process and product eval uation 
of most developmental projects* 

Some RDD&E activities are located In the Division of Instruction, es- 
pecially the district-wide testing program. Title t evaluation, curriculum 
development, and staff development* The Department of Planning, Research, 
and Evaluation functions, however, at least in a technical assistance 
role to RDD&E activities located outside the Division of Development and 
functions In a coordinating or supervisory role in relation to the evalua- 
tion functions In some of the projects within these programs. 
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Pennv^for Innovation Progran^ 

This program currently consists of more than thirty Innovative pro- 
jects funded from "a penny on the ta?: dollar.'* The program structure pro- , 
vMos for four proposaHgonerat fng groups— teachers, princlpalsi central 
office midole management, and executlvt* team management. Each group has 
a poor group screening committee wtilch passes approved projects to the 
stcoring committor for final approval, 

Porf prmancc Contracting 

Tv-fO major performance contracts totaling nearly one million dollars 
ore presently operational in five high schools, one junior high school, 
and one ciementary school, RFP's are near completion on several others 
for vocational^technical education in the new twenty*one million dollar 
Career Development Center* 

Dunbsr Cofnmqpitv LGarniM._£enter 

An elementary school in the inner-city area has been converted into 
a development/diffusion complex. In addition to the regular teaching 
and administrative staff, plus Title I programs, approximately one*ha)f 
million dollars annually from private foundation funds (H,Ross Perot 
Grant) has been used to provide an overlay development/diffusion staff of 
approximately thirty professionals plus support personneli organized 
around eight special centers, e^g.i language arts, mathematlcsi health 
and physical education, social studies, etc* This center has been 3 field 
test site for several developmental programs from two USOE regional educa* 
tional facilities and for several special Instructional products, such as, 
fPI mathcniatics. Sesame Street, Taba StrategieSi etc. 

Major Staff Chara£t<^rl^tics 

The General Superintendent of the District, Dr, Nolan £stes, is 
strongly supportive of RDD&E activities, as are.. the majority of the 
present Board of Education* The Associate Superi ntendent of Developmenti 
Mr, Rogers Barton, has had extensive esiperlence in proposal generation, 
instructional management, Innovative learning systems development, and 
project planning and administration* Two assistant superintendents are 
positioned in th3 Division—one to head the Department of Planning, 
Research, and Evaluation, and thj other to work with the Innovative Learning 
Systems Projects, especially in performance contracting project's. Both 
ciSsistant superintendents hold terminal degress and have experience in RDD^E 
act i vit i es. 
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The Division of OcvolopmcnC has nine professlonali with terminal dtigrees 
distributed by major field as follows (note: ono person holds tm doc- 
Corotcst Chcrafore, tho CoCat of ten); 



t. Educational Administration 2 

2. Educational research 2 

3. ElcmonCary education 2 

4. Educational psychology 1 

5* Sociology..* 1 

6. Curriculum development ^ I 

7. Philosophy # I 



Throe other professionals arc well along In doctoral programs* ai^ cf six 
vacant positions yet to be filled In the 1970-71 fiscal year, at least 
two are expected to have completed or be near completion of terminal de- 
grees. Appro;itmately 75 per cent of the remaining professionals, ex- 
cluding computer programmers, data systems anelysts, etc., have at least 
masters' degrees. The total authortzed manning level for the Division 
Is in excess of fifty professionals ptu^ support personnel. 
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VITA 

Faculty <ff EdugatlonAl Develppmont 

The Acad^imlc Faculty of Educattonol Ocvelopmont Is a program level 
organizational unit (departmental) of the Collisge o"^ Education of The 
Ohio State University* As such, It has responsibility for Ph*D* and 
H.A* programs In Educational Development and an undergraduate minor 
program in Educational Research* In addition to Its specified r^^s*- 
ponsibitity for offering these instructional programs, It Is the ad** 
ministrativc home of various funded research and development activities 
of Its members and for two centers, the Evaluation Center directed 
by Oantet L* Stufflebeam and the Program Management Center directed 
by Desmond Ccx>k* 

The graduate level programs arc designed flexibly to provide a 
basic comnvon core of experiences for students In the areas of systems 
analysts in education and empirical research methods* Beyond that 
core, individual students design programs using the entire resources 
of the university to prepare themselves In one or more of the 
following specializations: 

Educational Evaluation 
Educational Planning 
Educational Research Methodology 
Educational Program Management 
Instructional Technology 
Educational Change 

Students also are encouraged to devise programs which Include a substan** 
tial amount of work in related professional areas, most frequently ed* 
ucational administration or curriculum and Instruction* 

The undergraduate minor program is an Innovative effort Involving 
thirty quarter hours of credit In courses, seminars and internship 
setti ngs i ntended : 

I* To prepare some persons who will ultimately become teachers 
to a level of proficiency in Inquiry methods such that they will be 
able to use research as a basis for their own professional performance 
and growth* 

2* To prepare some undergraduates to fill occupational roles in 
evaluation, research, and areas In public school agencies* 

3* To provide a pool of trained undergraduates, some of whom may 
go on for graduate work in the field* 

The centre! focus of alt programs In this Faculty is directed at 
improvement, through empirical and systemic methods, of the way 
educational services are provided to learners in the real world* 
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Because of the rapid grovjth of too):, and roles for which persons may 
be prepared in this general area, the total program is and must remain 
open-'cnded, experimental, ond dynamic* Consequently, it H able to 
provide on appropriate academic home base for the efforts of the con- 
sort iu:n» 
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VITA 

Evaluation Contor 

The Evaluation Center, an agency of The Ohio State University College 
of Education^ ;$ conimitted to advancing the science and practice of educa* 
tional evaluation* To serve this purpose, the Center's Interdisciplinary 
team engages tn research^ development, Instruction^ leadership^ and eval*- 
uation service activities* 

The origin of the present Center traces back to the establishment of 
The Ohio State University Test Oevelopment Center in ISSl* Because of the 
urgent need for a more comprehensive approach to evaluation than that 
afforded by standardized testing^ the Test Oevelot^ment Center was expanded 
in 1965 into the present Evaluation Center^ which has been concerned with 
many modes of evaluation In addition to standardized testing* 

The Center has been engaged In five basic activities* The first ts the 
development and verification of theory In the area of educational evaluation* 
Many theoretical papers^ articles^ and several books have had thetr origin 
In the work of the Center* Also^ personnel of the Center have conducted 
a considerable amount of empirical research to test ^ew theory formulations 
in evaluation* The most notable accomplishment of this activity has been 
the development of the CIPP evaluation model which has been the subject of 
study, instruction^ and application on a national scale* 

The second activity is the development of materials and mechanisms for 
the implementation of evaluation* This activity includes the development 
of evaluation instruments and of institutional units for the Implementation 
of evaluation* The Center has developed mora than one hundred standar* 
dized educational achievement tests, including eight forms of the General 
Educational Oevelopment tests* l/ork has been performed for local* state, 
and national educational agencies in the development of evaluation systems* 
Center personnel have performed work of this nature in the Bureau of 
Elementary and Secondary Education and the Bureau of Educational Personnel 
Oevelopment in the U*S* Office of Educ^ition^ In the Southwest Educational 
Development Corporation in Austin, Texas» and in the Saginaw, Michigan and 
Columbus, Ohio Public Schools* The Center*s most notable past achievement 
In this area was the development of the Office of Evaluation and Research 
In the Columbus^ Ohio Public Schools* 

The third activity Is training* The Academic Faculty of Educational 
Development^ which i$ the organizational home for the Evaluation Center 
within th;,* College of Education, offers a Ph*0* program with a speciali- 
zation in evaluation* This specialization is supported mainly by the 
professorial staff in the Evaluation Center* The Center has conducted 
two Education Professions Oevelopment Act training institutes in evalua* 
tlon for groups of EPOA project directors* Also^ the staff has developed 
several simulation exercises to support training In evaluation* Past 
Ph*D* graduates from The Ohio State University with expe'*Ience in the 
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Evaluation Center and specialized training In evaluation Include Thomas 
R» Owens» Assistant Professor, Hawalf Curriculum Cwter; Howard 0» 
Merrrman^ Director, OffiCs; of Evaluation and Research, Cotutnbus Public 
Schools; Blarnc ft. Worthcn, Co^^Olrector, laboratory of Educational Re-* 
search, Unlvers ity of Colorado; James Heck, Oean« College of Education, 
University of Delaware; Jack M* Ott, Assistant Professor, The Ontario 
Institute for Studies In Education, Toronto, Canada; Bernard H» Qarbadora, 
ftesearch Associate* Program ftesearch and Design, Cincinnati Public Schools; 
Mfchac) H» Kean, Assistant tothe Superintendent, Phtladelphtd Public 
Schools; Hichaet S» Caldwell, Associate Dean, School of Education, Unl-* 
vcrslty of Virginia; Walter Caltngcr, Assistant Professor, University 
of Nebraska; and Michael 0* Hock, United Community Council, Coltmibus^ 
Ohio. 

The fourth activity pertains to the dissemination of Information 
concerninq educational evaluation* Means for dissemination that are 
employed by the Center Include the distribution of working papers; 
publication of articles In professional Journals, monographs^ and bocks; 
presentations at national meetings; and the conduct of special national 
conferences pertaining to evaluation* The Center personnel coordinated the 
publication activities of the Phi Delta Kappa National Study Committee on 
Evaluation and conducted the llth Phi Delta Kappa Symposium on Educational 
ftesearch (which had as Its focus the CIPP evaluation model)* 

The fifth activity Involves the conduct of evaluation studies* 
Evaluations have been performed In relation to school programs in 
Saginaw, Mtchlgna and Columbus, Ohio; a migrant education program In the 
Southwest Educational Development Corporation; and programs of The Ohio 
State University* These evaluation services provide a laboratory for re- 
search on evaluation and expert ence-ba^ed training* 

Collectively the five activities conducted by the Evaluation Center 
should provide a research training laboratory for the proposed consortium* 
based program* The Evaluation Center certainly will respond to some of the 
evaluation needs of other agencies In the consortium which will produce 
further training opportcrJties* 



no 



106 



VITA 
Phi Delta Kggpa 

The activities undertaken through the Research Service Center during 
the past year are reported below under the two areas of concentration in* 
itially established at the Memphis Qlonnial Council and endorsed at the 
San Oiego Gienni al ^Counc i 1 , dissemination of research results and resertrch 
training* 

D rssemi not ion 

Four activities encompass the Center*s dissemination efforts? The 
School Research Inforniotlon Service; Annual Research Syniposlaj Research 
Studies In Hducatlon; and Selected Study Committees* 

The School Researc h [ nf Qrmatlon Service * in Its first year of opera- 
tion SRIS accumulated approximately 500 reports for, Its library, encoded 
another 6,000 ERIC reports, and responded to kOO requests for Information* 
An SRIS search now covers a library collection of nearly ^0,000 documents. 
Ouring the past academic year responses were made to about 200 requests 
for Information per month. This is an increase of 80 responses per month 
(or c 67 per cent increase) over the previous year's responses* 

Two changes in the operation of SRIS are currently being structured. 
One is a change in response format, the other Is an additional service. 
Responses to requests may at present include* (1) copies of documents 
in the SRIS collection; (2) lists of document numbers for the relevant 
documents in the ERIC collection; and (3) directions for ordering the 
ERfC documents* At the present time a file of abstracts of the ERIC 
documents is being created* V/hen that file is brought up to date the re- 
sponse will consist of copies of relevant ERIC documents* and Instructions for 
ordering ZRiC documents* This change should make the response to a request 
more informative than we are now able to make It* 

The additional service is a modified selective dissemination service* 
Lists of descriptor terms have been generated for several professional 
specialities In education* If fifty persons wMl subscribe for a specific 
specialty list* SRIS can offer a quarterly service In which th;; subscriber 
will receive a publication t'^at presents abstracts of all documents added 
to the retrieva* system that are relevant to that area of specialty* A 
sample of such a publication is new being prepared to be used to promote 
subscriptions* 

Annual Resea'-ch Svmoosium^ The publication for the ninth annual 
symposium* Dayeslan Stati£;tlcs has been evailable for several months* 

E* Peacock, Publishers* Indicated that they filled approximately 1,000 
requests for this publication as it first became available* Reviews of this 
manuscript have not appeared in the professional literature^ as yet, how*- 
cver* one is scheduled for a coming issue of the Journal of Educational 
Measurement* Wlion that Is available copies will be sent lo members of the 
Doard of Olrectors* 



111 



J 07 



Tho manuscript has not boon completed for the tenth annua) sYmpcsIum* 
As Indicated oarlicr a variety of problems plagued this syniposlum* Two 
Outstanding papers were developed for It and an outline for a third* Pub* 
licatlon of a manuscript on this symposium has been deferred until: (l) 
the third paper Is completed; and (2) careful critiques of all three have 
been v/ritten by competent Individuals. Tho death of the wife of the 
chafrmon of this symposium, Asahel Vioodruff, has been a factor In the delay* 

The eleventh annual annual symposium was held at The Ohio State Untver^ 
sity, June 22-^2^, 1970* Approximately 65 persons participated In this 
session* In preparation for the symposium all participants were sent copies 
of the manuscript, Educational Evaluation and Oecis Ion*Hakln9» prepared by 
the Phi Delta Kappa Study Committee on Evaluation (a Ski page manuscript 
currently being published by E* Peacock* Publishers)* SI:c of the par* 
ticlpants were asked to prepare a critique of the evaluation theory and 
methodology presented In that manuscript from the point of view of their level 
of operation (They represented public school systomSt colleges, research 
and development agencies* state departments of education, and the federal 
government)* 

pesearcj} Studies In Educat^^n . The volume reporting dissertations In 
education completed and underway In the ld67*-8 acadetnic year was completed 
a^d made available* Vfork on 1963-^9 Is well underway* Carter Good has sub*- 
mltted the research methods bibliography and Joe Hapes reports a target 
date of September I for the completion of the collation and Indexing tasks* 

Some conversation has been Initiated regarding the facilitation of this 
task through the accfuisttton of tapcdrlven I8H typing equipment* This 
er;ulpment will not only Improve the appearance of Research Studies In 
education but It will also expedite the preparation of camera ready copy* 

Study CommtttoGS ^ Progress has been made during the year on tho part 
of two of the three s:/jdy committees started through recommendations of the 
Research Advisory Canmtttee* Under the leadership of Daniel Stufflebeam 
tho Study Committee on Evaluation has developed the manuscript described 
in connection with the eleventh annual research symposium* Other persons 
participating In the Vfor< of this study committee are Ors* Egon 6* Cuba 
(Indiana University), Walter Foley (University of Iowa), Robert Hammond 
(Ohio State University), Halcolm Provus (Pittsburgh Public School)* 
Howard Herriman (Columbus Public Schools),, and tH 1 1 lam J* Gephart* The 
manuscript is now tn the hands of the publisher, T* E* Peacock and a target 
publication date of January, 1971 has been set* 

RGsearch Trainino 

The Research Service Center's efforts relating to research training 
IncludG<J; (I) Conduct of the ftetional Symposium for Professors of Edcca* 
tfonal Research: 1969; (2) Planning of HSPER: 1970; (3) Participation In 
the AERA Task Force on Research Training; (k) Conduct of a USOE Research 
Project on the application of the Convergence Technique to basic studltss 
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or reading; (5) Chairmanship of the AERA Special Interest Group! Profes- 
sors of Educational Research; and (6) Study of the institutions and pro- 
grams offering the doctorate in sclucatlon* 

Additional Activltios 

The AERA-PDK Research Av/arcJ Committee selected Dr# Benjamin Bloom, 
University of Chicago^ as recipient of the 1970 av/ard for his work on sta* 
bility and change in human characteristics* He wi\} present a paper on 
his v/ork at the February meeting of AERA and Pht Delta Kappa will have 
publication rights on thot manuscript. 

At request of ,he National Center for Educational Rosoorch and Devel- 
opment of the U S Office of Education, The Director of Research Services 
participated in the writing of requests for proposals for two programs: 
(i) the three projects proposed as phase I of tbe RSO program on reading; 
anC (2) projects for the systematic development of materials to be used in 
the training of research^ development, diffusion and evaluation personnel. 

There have been numerous requests for copies of the Research Service 
Center's Occasional Papers* Tiik> changes have been made In this series; 
(1) the or.isting papers have been redone using the new IBH equipment (This 
has been done as existing copies have been depleteo); (2) papers developed 
by persons outside the Research Center have been selected for this series 
(this typically is a paper that Is too long for a journal article and which 
contains information related to the two general directions of the Research 
Service Center), The current list Includes: 

K l/illtam J* Gephart - The Problem £^ Problem Delineation Techniques 

2, Bruce B. Bartos * A Review of Instruments Developed to be Used In 
the Evaluation of the Adequacy of Reported Research 

3. Uiliiam J, Gephart - Profiling Educational Research 

William J, Gephart - Application of the Convergence Technique 
to Reading 

5, William J# Gephart - The Convergence Technique and Reading; A 
Progress Report 

6, William Gephart - The Eight General Research Methodologies: 
A Facet Analysis of the Research Process 

7, William J# Gephart 8- Bruce D, Bartos * Profiling Instructional 
Package 

3, Gene G, Glass - Educational Knowledge Use (currently being printed) 

9, Warren G, Ftndiey - Measurement and Research In the Service of 
Edixatlon (currently being printed) 



113 



t/nilam J, Gephart 



109 



VtTA 

£ itv of ^aqlna w Publ ic JScHoqIs System 
Tho PeoartinQnt of TostinQ and gvaluatton 

The Department was established In the Fall of 1S6Q# A staff of five 
professionals from various fields In educatJoci Is retained. The staff is com- 
prh^ed of a director and four cvaluators* 

Since Its Inception the Department of Testing and Evaluation has been 
affiliated with The Ohio State University Evaluation Center for purposes of 
obtaining post^-graduate training in CIPP evaluation methodology and con* 
sul tant services. 

Presently the Department of Testing and Evaluation Is involved In all 
major evaluations, (local, state^i^and federal) which ere either required by 
law or mardated by the Superintendent* Some of these evaluation activities 
arc presented in the list below* 

Current f^ctivities of the Department of Testing and Evaluattoni 

I* Evaluation of Project BRiTE 

a* Evaluation of the Instructional Improvement Program 

at Houghton School 
b* Evaluation of BRITE pilot programs and dissemination 

activities 

2* Evaluation of ESEA Title I Programs 
a* Language Master Program 
b* Continuation School Program 
c* Parochial School Programs 
d* Pre-School Programs 

3* Implementation of the Michigan Department of Education's 

(Ourcau of Research) Evaluation Design for Selection 3 Programs 

^* Implementation of the Michigan Assessment Program 

5# Administration of the Standardized Testing Program 
a* Conducting In-scrvlce Education sessions relative 
to implementation and utilization test data 

6# Consultation with those involved in curriculum planning 
throughout the school system (building principals. Joint 
Curriculum Coffwnittee Members, etc*) 

7* Reporting to state and federal agencies 

C# Statistical analysis of test data 

9* Focusing ano development of evaluation designs 

10* Report preparation and dissemination 
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The Ocpartmcntt in addition to the above* has developed a rather 
cocnplct^ set of working policies whtch have been approved by the Soord 
of Education* The Department has developed an Instrument bank (^25 
titles), Innovative practices bank* data bank* and professlonat library 
in the areas of evaluation* research* and currtculum development* Orlef 
resumes indicating the leadership capabilities of the professional pcrsoni^el 
involved tn evaluation and research follow; 

Dr. Jack P. Taylor » Presant Position: Superintendent 

Academic backflroup^i &*Sc* In Education* Bowling Green State University* 
1953; H.Sc. In Education* Bowling Green State University* 195^; Ph.D.* 
The Ohio State University* 1966. Pufalj^ tl g pS! The Ohio School Boards 
Journal 1964* 1965; The MUhlgan School Boards Journal 19^; School 
Management I97O, Exoorlgnce t Curriculum Coordinator* high school principal* 
research assistant* seminar coordinator* and Superintendent. Hmum^: 
E. 2, lev/Is Av/ard* The Ohio State University; The Worth HcClure Scholar- 
ship In School Administration* AAjSA and the NEA; t/ho*s Who In American 
Education and t/ho's V/ho In the Midwest. State and national C«tiq)Ittees ! 
Program of the American Association of School Administrators* Press and 
Evaluation Committee of the American Association of School Administrators* 
College Admission Convnittee of the Michigan Association of School Admin* 
Istrators. 

James A. Adams — Prosnnt PosTtlon! Assistant Superintendent for Instruction 

Academic backgr o^ fp ^t A.B.* In Secondary Education* Horehead State Unlver* 
sity I95G; n.A. tn School Administration* Morehead Stete University \S6\; 
Ph.D. (admitted to candidacy) The Ohio State University 1970. Publications! 
The Ohio School Boards Journal* 1970; ^'The Implementation of Evaluation 
Theory'* in Report of the Eleventh Annual Phi Delta Kappa Symposium on 
Educational Research (to be published), gayer] f pee ! Elementary teacher* 
junior high teacher* elementary princlpol* junior high principal* Admlnls-* 
trative Assistant to the Superintendent* Research Associate* Assistant 
Superintendent for In. 'ruction. Hffpprai E. E, lewis Award In School Ad* 
ministration* The Ohio State University. 

Dr. H. Douglas CAvInotcn " Present Poslttont Assistant Superintendent 
of Edurattonal limrovement 

Academic baylyjrognd ? B«s. In Elementary Education* Central State Collage* 
1957; H.A. in School Psychology* The Ohio State University* 1958; Ph.D.* 
1966* The Ohio State University. Exoerl^nce i Elementary teacher* coar** 
dinator of standardized testing program* recreation director* school psy 
chologlst* supervisor of testing and research* educational consultant* 
Assistant Superintendent. Honi;>ry 1 Jessie Smith Noyles Foundation 
Academic Scholarship, gtatj ^^ gnd Matlonal Comnlttees ! Indiana State Board 
of Education and Cocwnlsston on Textbook Adoptlon*~^T967~68* 
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Dr. Richard KeHev " PnQsent Pos tt ion: Assistant Sqt)erintef:den t 

Academic backoround ! B,S# In Business AdminIstratIon» The Ohio State 
University^ 1952; B.S. In Education, The Ohio State UnWorsIty, (956; M.A. 
In Educational Administration, The Ohio State Unlve sity, (959; Ph# 0, In 
Educational Administration, The Ohio State University, I962# 
Pu blic.-^tions t Ohio Schools* l9o3, I5SV, The Executive Housekeeper^ IS69. 
experience : Distributive Education Coordinator^ Executive Head, Research 
Assistant, Assistant Superintendent* 

Jcrrv Baker PreseQt P^stttoa; Otrector of TestJjtq and Evaluation 

Academic backgypi^rid ! B#S#, M#A#, Ed#S., Central Michigan University, .1 
Publ I cations ! Michigan School Board Journal, 1967* Experien ce: Teacher, 
Research Assistant^ Senior High School Guidance Counselor, Title I Evaluation 
CoordI(.ator » Director of Testing, Director of Testing and Evaluation, 
State and National Michigan Dral Appraisal Board* 

The following description outlines further some of the characteristics 
of the school system and the community along with some evidence of the school 
system's financial strength* 

Streamline Operations 

The City of Saginaw is an industrial oDfTsnunity with a population cf 
approximately 1DO,DO0* The population Is changing In racial balance following 
the national .rend, ue*, the white middle class and upper class population 
is moving to suburban areas and being replaced by minority races and poor white 
peopi e* 

The schools serve approximately 23,000 pupils* There are 28 elementary centc 
6 junior highs, 2 high schools and 1 vocational school* Test results Indicate 
that the ghetto schools have not been successful In meeting the needs of 
children in those areas. The technical school has also fallen short of the 
needs of pupils for this type of program* 

Saginaw relies on the property tax for slightly over half of Its local 
support. The total tax duplicate Is over $'t5D,DDD,0DD or appro;cImteIy 
$I9»6do per pupil* The present ta;; rate for operating purposes ts 23.8 mills* 
An additional 2 mills Is used to retire bonds which are used to fund capital 
projects* The State of Michigan provides approKlmQtely 40% of the operating 
revenues of the a'istrict* 

The school has budgeted $I9,I35»DDD for tha school year l97D-7'» or 
approximately $333 per pupil* Instruction alom; will run nearly $15 million 
dollars or appro;;imately $652 per pupil* These <^igures do not include nearly 
$7DD,OD0 in federc) monies or $70,000 from other special grants* 

In 1567^ the taxpayers providecJ a new levy of 0 mills to support a total 
educational program which Included an Inter-clty program called BRtTE and 
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and new vocational technical school to be called the Career Opportunity Center. 
The former has been in operation for two years and the latter wIU open ii> the 
fall of 1972. 

Funds to support theie programs have been put Into reserve to assure that 
they will have adequate funds to Implement them as they are developed. At 
the time of this writing $1,338,000 are in those reserve funds. 
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INTERVIEW SCHEDULE 
Development of the Model Training Projects Consortium 



Section I 

L What were the needs of your organization (relate to): 

a) Problems 

b) Projects 

c) Organizational (plans) 

2. What problems were associated with the needs cited above? 

3* How did evaluation effect decisions which related to the resolution of 
needs and problems? 



Section 1 1 

t. What needs ^ problems, and opportunities Influenced your decision to 
become a member of the consortium? 

2. What problems did you experience in getting a letter of commitment 
from your agency? 

3. In revi^jwing the criteria for selection (hand interviewee copy of 
criteria) which criteria was the most difficult? Why? 



Section 1 1 1 

1. What advantages did you expect to gain as a member of the con*,ortfUm? 

2. What were the advantages of belonging to the consortium? 

3. What problems did you experience as a result of belonging to the 
consortium? 

^« To what extent was your agency involved with other members of the 
consortium? 

a) Sharing of materials 

b) Working of resolution to conunon problems 

c) Exchange of personnel 

5* Were the site visits b/ the MTP staff of value to your organization? 
I n what way? 

6. Did y^u participate In any site visits? 

7* To what extent v/ere you or members of your staff Involved with the MTPs 
evaluation unit'' 

8* In what ways did the PRDB serve your agency? 
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5. Old you or members of your organization work with the PROB stoff? 

10. Were you or members of your staff involved in shared activities with 
the materials development component of the model training project? 

Section IV 

I . To what extent were you and your agency involved in the planning phases 
of the MTP project? 

2, O'd you participate in the selection of members of the consortium? 

Were you satisfied with your opportunity to take part In mTP decision- 
making? 

Section V 

I* How many Internships were available In your organization for hTP students? 

2, What problems did you fac^ in acquiring Internships? 

3* What problems did you experience in relationship to the advisement of 
Interns? (example: Information concerning problems you experienced) 
if answer to one was no, go to next s^^ctlon* 

^, Were you satisfied with the Interns you received from the MTP? 
Section VI 

1, What was the total operating budget of the evaluation unit as of 
December 30, 1972? (If no budget was available, go to next section) 

2, How much of the money was soft money? 
:>ect ion V 1 1 

1, Oid your organization Install an evaluation component? (if no unit was 
established go to next section) 

2. To what extent was the evaluation component developed? 
3* How many people were involved In your unit? 

^, What problems did you experience in establishing the unit? 

5. What solutions were reached? 

6. How much internal training was necessary in order to implement your 
evaluation plan? 
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CRITERIA FOR HTP CONSORTIUM MEMBERSHIP 

CRITERIA FOR FIELD-BASED TRAINING AGENCIES PARTICIPATION IN 
THE ONTO STATE UNTvIRSITY-BASED MODEL TRAINING PROGRAM IN EVALUATtOH 

)* The Agency ts developing or is planning to irvsti tutional ize an 
evaluation unit <n tts organization* 

2. The agency shall sobmit a plan in writing for acccmpl ishing the 
institutionalization of evaluation* This plan will include: 

a) functions of the evaluation unit 

b) staff positions of the evaluation unit 

c) present personnel capabilities for filling positions In ^*b'^ 

d) personnel needs for unfilled position 

3. The agency shall commit resources to implement the institutionalization 
plan. Support by the agency for stafVIng^ for Installation of the 
evaluatlc^n unlt^ and for operation of the evaluation unit is estimated 
at $200^000 per year or some reasonable amount of the operating budget. 

The agency shall commit an on-staff person (Ph.D. level) as an adjunct 
professor of the Academic Faculty of the Educational Development 
Department at The Ohio State University to direct the on-site training 
in evaluation for trainees. This person shall commit himself to: 

a) conduct regularly scheduled seminars for trainees within the 
developed guidelines of the project 

b) attend all meetings and workshops scheduled for agency representatives 

5. The agency shall serve as a recruitment and placement agency for 
trainees of the program. 

6. The agency shall serve as an organization to field-test appropriate 
materials. 

7* The agency shall commit itself to pay support and to pay tuition 

costs for all trainees receiving Ohio State University credit In the 
time they are based as an Intern In that agency. 

8. The agency and the Ohio State University shall jointly recruit and 
train staff necessary to meet the manpower needs identified in the 
project and also the needf of the agency. 

9. Applicable only to those agencies so indicated In the approved budget, 
provide cost estimates up to $^,600 to provide replacement time for 
individual named In item ^ above and for other miscel lanecus costs. 
This reimbursement Is to be provided after the performance criteria In 
these Items N8 are met. 
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Provide a letter to Oarrell Root (614-422-1368) agreeing generally 
to these criteria and signed by an authorized business official of 
your agency along with his telephone number. This letter should 
list the individual named in item 4, and his telephone number and 
attach the plan requested in Item 2« An agreement will be submitted 
for your review as soon as we receive your letter and obtain 
Government approval. 
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